From a0e850a9f15b6e750509c788ee84b03d8c770c95 Mon Sep 17 00:00:00 2001 From: Malte Ubl Date: Tue, 20 Oct 2026 13:41:54 -0700 Subject: [PATCH 01/51] Share one listener list between typed and untyped uses of an EventEmitter An emitter used both through its static type and through an untyped reference aborted: a typed `on` followed by an untyped `emit` (or a typed `emit` from a class body reaching a listener added through `any`) hit "computed event collided with a fixed event", and an untyped `emit('error', ...)` trapped outright. Listener entries held either a typed adapter or a checked-dynamic thunk, and the emit core trapped when an entry lacked the form the emit carried. Entries may now carry both forms. Listeners registered through scr_emitter_on_dyn keep their thunk beside the typed adapter; typed emits call the adapter and untyped emits call the thunk with the arguments as given. The class dispatch of compiled emitters registers untyped listeners under an event typed code fixes with a typed adapter for its tuple, and converts the arguments of an untyped emit of such an event to the tuple for listeners with typed parameters (scr_emitter_emit_mixed; a mismatch throws the checked conversion's TypeError). Registrations and emits through untyped values otherwise go through scr_emitter_on_any/emit_any: runtime adapters under 'error' and the meta events, and Node's unhandled-'error' contract. The emit core throws instead of trapping when an entry has no usable form. The dynamic half of the registry sits behind a hook only on_any installs, and on_flex/emit_flex keep their plain behavior, so programs without untyped emitter calls link none of it. listenerCount, eventNames, removeAllListeners, listeners, rawListeners, setMaxListeners and getMaxListeners through an untyped value on a compiled emitter answered "is not a function"; they now dispatch to the native registry. Co-Authored-By: Claude Opus 5.5 --- .../limitations/builtin-apis/eventemitter.mdx | 5 + .../compiler/src/backend/llvm/lib-process.ts | 53 ++- .../compiler/src/backend/llvm/lib-shared.ts | 3 + .../lowering/class-dynamic-dispatch.ts | 134 +++++- .../src/frontend/lowering/lower-calls.ts | 6 + .../frontend/lowering/lower-event-emitter.ts | 237 +++++++++- packages/compiler/src/ir/builtin-effects.ts | 4 + .../compiler/src/ir/builtin-signatures.ts | 15 + packages/compiler/src/ir/validate.ts | 20 +- ...80-emitter-typed-untyped-listeners.ts.json | 6 + ...581-emitter-untyped-error-and-meta.ts.json | 6 + ...-emitter-class-expression-escapes.cjs.json | 6 + .../6583-emitter-untyped-surface.ts.json | 6 + ...itter-class-body-untyped-listeners.ts.json | 6 + packages/runtime/src/scr_events_emitter.c | 407 ++++++++++++++++-- packages/runtime/src/scr_events_emitter.h | 28 +- .../6580-emitter-typed-untyped-listeners.ts | 50 +++ .../6581-emitter-untyped-error-and-meta.ts | 59 +++ .../6582-emitter-class-expression-escapes.cjs | 61 +++ tests/corpus/6583-emitter-untyped-surface.ts | 47 ++ ...84-emitter-class-body-untyped-listeners.ts | 54 +++ 21 files changed, 1148 insertions(+), 65 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6580-emitter-typed-untyped-listeners.ts.json create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6581-emitter-untyped-error-and-meta.ts.json create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6582-emitter-class-expression-escapes.cjs.json create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6583-emitter-untyped-surface.ts.json create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6584-emitter-class-body-untyped-listeners.ts.json create mode 100644 tests/corpus/6580-emitter-typed-untyped-listeners.ts create mode 100644 tests/corpus/6581-emitter-untyped-error-and-meta.ts create mode 100644 tests/corpus/6582-emitter-class-expression-escapes.cjs create mode 100644 tests/corpus/6583-emitter-untyped-surface.ts create mode 100644 tests/corpus/6584-emitter-class-body-untyped-listeners.ts diff --git a/docs/content/limitations/builtin-apis/eventemitter.mdx b/docs/content/limitations/builtin-apis/eventemitter.mdx index a9f03b857..0e085c56f 100644 --- a/docs/content/limitations/builtin-apis/eventemitter.mdx +++ b/docs/content/limitations/builtin-apis/eventemitter.mdx @@ -1,5 +1,10 @@ - **EventEmitter:** Literal event names use fixed argument tuples. Computed string names require a constant prefix or suffix that excludes `error`, meta events, and stream-internal events; their arguments must match the exact arity and support checked conversion. Unrestricted names, symbols, names that may reach internal events, and typed `listeners()`/`rawListeners()` results for names with possible computed registrations are unsupported. + An emitter reached through an untyped value shares one listener list with its typed uses, so untyped registrations and emits reach typed listeners and typed emits reach untyped listeners. + An untyped `emit` of an event that has a listener with typed parameters converts its arguments to the event's typed tuple before any listener runs, and throws a `TypeError` when they do not convert. + Through an untyped value, `listeners()` and `rawListeners()` of an event with a listener that has typed parameters are unsupported, and `emit('error')` of an object that is not an Error with no `error` listener is unsupported. + Untyped `newListener` and `removeListener` listeners receive `undefined` in place of a listener registered with typed parameters. + Reading EventEmitter methods as properties through an untyped value (`typeof emitter.listeners`) answers `undefined`. `events.once` accepts EventEmitter and stream instances with a literal event name and an optional object-literal `{ signal }`; a stream's `data` event, other options values, and EventTarget emitters are unsupported. Reading `defaultMaxListeners` through a CommonJS destructured or member binding (`const { defaultMaxListeners } = require("events")`) is unsupported. diff --git a/packages/compiler/src/backend/llvm/lib-process.ts b/packages/compiler/src/backend/llvm/lib-process.ts index 097fcb4de..4cee7c1dd 100644 --- a/packages/compiler/src/backend/llvm/lib-process.ts +++ b/packages/compiler/src/backend/llvm/lib-process.ts @@ -852,7 +852,7 @@ export function emitErrorsEventsLibCall(host: LlvmEmitterContext, e: LibCallExpr host.emitPendingCheck(); return { name: t, type: e.type }; } - if (e.fn === "emitter.emitFlex") { + if (e.fn === "emitter.emitFlex" || e.fn === "emitter.emitAny") { const args = e.args.map((a) => host.emitExpr(a)); const n = args.length - 2; let payload = "null"; @@ -866,10 +866,57 @@ export function emitErrorsEventsLibCall(host: LlvmEmitterContext, e: LibCallExpr } payload = arr; } - host.declare(`declare zeroext i1 @scr_emitter_emit_flex(ptr, ptr, ptr, ${host.sizeType})`); + const sym = e.fn === "emitter.emitAny" ? "scr_emitter_emit_any" : "scr_emitter_emit_flex"; + host.declare(`declare zeroext i1 @${sym}(ptr, ptr, ptr, ${host.sizeType})`); const t = B.tmp(); B.line( - `${t} = call i1 @scr_emitter_emit_flex(ptr ${args[0]!.name}, ptr ${args[1]!.name}, ptr ${payload}, ${host.sizeType} ${n})`, + `${t} = call i1 @${sym}(ptr ${args[0]!.name}, ptr ${args[1]!.name}, ptr ${payload}, ${host.sizeType} ${n})`, + ); + host.emitPendingCheck(); + return { name: t, type: e.type }; + } + if (e.fn === "emitter.emitMixed") { + // (recv, name, k, ...k dyn arguments, ...converted tuple): the dyn + // arguments ride an exact-length array like emitFlex, the tuple the + // pointer-classed variadic tail like emitter.emit. Borrowed; may throw. + const count = e.args[2]!; + const k = count.kind === "numLit" ? count.value : 0; + const args = [e.args[0]!, e.args[1]!, ...e.args.slice(3)].map((a) => host.emitExpr(a)); + let payload = "null"; + if (k > 0) { + const arr = B.slot(); + B.entryAllocas.push(`${arr} = alloca [${k} x ptr]`); + for (let i = 0; i < k; i++) { + const slot = B.tmp(); + B.line(`${slot} = getelementptr inbounds [${k} x ptr], ptr ${arr}, i64 0, i64 ${i}`); + B.line(`store ptr ${args[i + 2]!.name}, ptr ${slot}`); + } + payload = arr; + } + const tuple = args.slice(2 + k).map((a) => { + const ty = host.llType(a.type); + if (ty === "double" || ty === "i1") { + const slot = B.slot(); + B.entryAllocas.push(`${slot} = alloca ${ty} ; EventEmitter scalar argument`); + B.line(`store ${ty} ${a.name}, ptr ${slot}`); + return `ptr ${slot}`; + } + return `ptr ${a.name}`; + }); + host.declare( + `declare zeroext i1 @scr_emitter_emit_mixed(ptr, ptr, ptr, ${host.sizeType}, ...)`, + ); + const t = B.tmp(); + B.line( + `${t} = call zeroext i1 (ptr, ptr, ptr, ${host.sizeType}, ...) @scr_emitter_emit_mixed(` + + [ + `ptr ${args[0]!.name}`, + `ptr ${args[1]!.name}`, + `ptr ${payload}`, + `${host.sizeType} ${k}`, + ...tuple, + ].join(", ") + + `)`, ); host.emitPendingCheck(); return { name: t, type: e.type }; diff --git a/packages/compiler/src/backend/llvm/lib-shared.ts b/packages/compiler/src/backend/llvm/lib-shared.ts index 8cd40240f..007c78c9e 100644 --- a/packages/compiler/src/backend/llvm/lib-shared.ts +++ b/packages/compiler/src/backend/llvm/lib-shared.ts @@ -1050,6 +1050,7 @@ export const LIB_FN_SYMS: Record = { "emitter.off": "scr_emitter_off", "emitter.offDyn": "scr_emitter_off_dyn", "emitter.onFlex": "scr_emitter_on_flex", + "emitter.onAny": "scr_emitter_on_any", "emitter.checkListener": "scr_emitter_check_listener", "emitter.removeAll": "scr_emitter_remove_all", "emitter.emitError": "scr_emitter_emit_error", @@ -1058,6 +1059,8 @@ export const LIB_FN_SYMS: Record = { "emitter.countDyn": "scr_emitter_listener_count_dyn", "emitter.names": "scr_emitter_event_names", "emitter.listeners": "scr_emitter_listeners", + "emitter.listenersDyn": "scr_emitter_listeners_dyn", + "emitter.hasTyped": "scr_emitter_has_typed", "emitter.setMax": "scr_emitter_set_max", "emitter.setMaxChk": "scr_emitter_set_max_chk", "emitter.setDefaultMaxChk": "scr_emitter_set_default_max_chk", diff --git a/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts b/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts index de84e3fed..c9e6768a6 100644 --- a/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts +++ b/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts @@ -68,7 +68,11 @@ import { isClassCallback } from "./class-callbacks.js"; import { CaughtValueDispatch } from "./caught-value-dispatch.js"; import { refreshDescriptorGuards } from "./class-descriptors.js"; import { ClassConstructionDispatch } from "./class-construction.js"; -import { emitterRooted } from "./lower-event-emitter.js"; +import { + emitterRooted, + lowerDynamicEmitterEmit, + lowerDynamicEmitterOn, +} from "./lower-event-emitter.js"; import { protocolArm, protocolFieldArm } from "./class-protocol-dispatch.js"; import { STREAM_OPERATORS } from "./lower-stream.js"; import { deletedPropertyNames, fixedFieldFamily } from "./js-class-fields.js"; @@ -97,6 +101,24 @@ const EMITTER_METHODS = new Set([ "off", "removeListener", ]); +/** The rest of EventEmitter.prototype, answered through an untyped value + * on emitters that are not streams (streams keep their own surface). */ +const EMITTER_INTROSPECTION_METHODS = new Set([ + "listenerCount", + "eventNames", + "removeAllListeners", + "listeners", + "rawListeners", + "setMaxListeners", + "getMaxListeners", +]); +/** True when the class is or extends one of the runtime stream classes. */ +function streamRooted(info: ClassInfo): boolean { + for (let c: ClassInfo | null = info; c; c = c.base) { + if (c.builtinStream !== undefined) return true; + } + return false; +} const STREAM_BOOL_PROPERTIES = new Set([ "readable", "readableEnded", @@ -1223,6 +1245,13 @@ export class ClassDynamicDispatch { const matching = this.propertyReceivers(lowerer, true).flatMap(({ info }) => { if (!info || info.fields.has(method) || info.builtinError) return []; if (emitterRooted(lowerer, info) && EMITTER_METHODS.has(method)) return [info]; + if ( + emitterRooted(lowerer, info) && + !streamRooted(info) && + EMITTER_INTROSPECTION_METHODS.has(method) && + !findMethodOn(lowerer, info, method) + ) + return [info]; if (info.builtinStream) { return STREAM_METHODS.has(method) || (STREAM_OPERATORS.has(method) && info.builtinStream !== "w") @@ -5055,6 +5084,66 @@ export class ClassDynamicDispatch { return [store, { kind: "return", value: null, loc }]; } + /** EventEmitter.prototype's introspection and bookkeeping members on an + * emitter seen through an untyped value: Node's results over the native + * listener registry. */ + private emitterIntrospection(lowerer: Lowerer, dispatch: Dispatch, receiver: IrExpr): IrStmt[] { + const { method, loc } = dispatch.source; + const incoming = dispatch.source.args.map((_, i) => varRef(`p.${i + 1}`, DYN, loc)); + const ret = (value: IrExpr): IrStmt[] => [ + { kind: "return", value: lowerer.coerceToExpected(value, DYN), loc }, + ]; + const call = (fn: IrLibFn, args: IrExpr[], type: IrType): IrExpr => ({ + kind: "libCall", + fn, + args: [receiver, ...args], + type, + loc, + }); + // Node indexes _events with the argument itself: an omitted name is + // the key "undefined". + const eventName = (): IrExpr => + incoming.length + ? lowerer.coerceToExpected(incoming[0]!, STRING) + : { kind: "strLit", value: "undefined", type: STRING, loc }; + switch (method) { + case "listenerCount": + return ret( + incoming.length >= 2 + ? call("emitter.countDyn", [eventName(), incoming[1]!], F64) + : call("emitter.count", [eventName()], F64), + ); + case "eventNames": + return ret(call("emitter.names", [], { kind: "array", elem: STRING })); + case "removeAllListeners": + return ret( + incoming.length + ? call( + "emitter.removeAll", + [eventName(), { kind: "boolLit", value: false, type: BOOL, loc }], + receiver.type, + ) + : call( + "emitter.removeAll", + [ + { kind: "strLit", value: "", type: STRING, loc }, + { kind: "boolLit", value: true, type: BOOL, loc }, + ], + receiver.type, + ), + ); + case "listeners": + case "rawListeners": + return ret(call("emitter.listenersDyn", [eventName()], DYN)); + case "setMaxListeners": + return ret( + call("emitter.setMaxChk", [incoming[0] ?? dynUndefinedExpr(loc)], receiver.type), + ); + default: + return ret(call("emitter.getMax", [], F64)); + } + } + private methodBody( lowerer: Lowerer, dispatch: Dispatch, @@ -5070,6 +5159,14 @@ export class ClassDynamicDispatch { loc, }, ]; + if ( + emitterRooted(lowerer, info) && + !streamRooted(info) && + EMITTER_INTROSPECTION_METHODS.has(method) && + !findMethodOn(lowerer, info, method) + ) { + return this.emitterIntrospection(lowerer, dispatch, receiver); + } if ( info.builtinStream || (emitterRooted(lowerer, info) && @@ -5090,18 +5187,37 @@ export class ClassDynamicDispatch { ]; if (method === "emit") { if (!incoming.length) return fence(); - return ret({ - kind: "libCall", - fn: "emitter.emitFlex", - args: [receiver, lowerer.coerceToExpected(incoming[0]!, STRING), ...incoming.slice(1)], - type: BOOL, - loc, - }); + const event = lowerer.coerceToExpected(incoming[0]!, STRING); + const bridged = streamRooted(info) + ? null + : lowerDynamicEmitterEmit(lowerer, receiver, event, incoming.slice(1), loc); + return ret( + bridged ?? { + kind: "libCall", + fn: "emitter.emitAny", + args: [receiver, event, ...incoming.slice(1)], + type: BOOL, + loc, + }, + ); } if (registering || removing) { if (incoming.length < 2) return fence(); const event = lowerer.coerceToExpected(incoming[0]!, STRING); const cb = incoming[1]!; + const bridged = + registering && !streamRooted(info) + ? lowerDynamicEmitterOn( + lowerer, + receiver, + event, + cb, + method === "once" || method === "prependOnceListener", + method.startsWith("prepend"), + loc, + ) + : null; + if (bridged) return ret(bridged); return [ { kind: "exprStmt", @@ -5113,7 +5229,7 @@ export class ClassDynamicDispatch { fn: registering ? info.builtinStream ? "stream.onDyn" - : "emitter.onFlex" + : "emitter.onAny" : "emitter.offDyn", args: [ receiver, diff --git a/packages/compiler/src/frontend/lowering/lower-calls.ts b/packages/compiler/src/frontend/lowering/lower-calls.ts index b1b4086cd..498e2359d 100644 --- a/packages/compiler/src/frontend/lowering/lower-calls.ts +++ b/packages/compiler/src/frontend/lowering/lower-calls.ts @@ -4019,6 +4019,12 @@ const DYN_DISPATCH_METHODS = new Set([ "prependOnceListener", "listeners", "listenerCount", + // The rest of EventEmitter.prototype, which compiled emitters answer + // through the class dispatch (class-dynamic-dispatch.ts). + "rawListeners", + "eventNames", + "setMaxListeners", + "getMaxListeners", "write", "end", "destroy", diff --git a/packages/compiler/src/frontend/lowering/lower-event-emitter.ts b/packages/compiler/src/frontend/lowering/lower-event-emitter.ts index f8588fe48..6b79666b1 100644 --- a/packages/compiler/src/frontend/lowering/lower-event-emitter.ts +++ b/packages/compiler/src/frontend/lowering/lower-event-emitter.ts @@ -53,6 +53,8 @@ import { BOOL, canBoxFuncIntoDyn, canConvertToDyn, + canDynCheckTo, + isDynTypedRefType, DYN, F64, type IrExpr, @@ -69,7 +71,7 @@ import { VOID, } from "../../ir/ir.js"; import { STREAM_FORCED_EVENT_NAMES, streamForcedTuple, streamSidesOf } from "./lower-stream.js"; -import { boolLit, strLit, varRef } from "../../ir/build.js"; +import { boolLit, dynUndefinedExpr, strLit, varRef } from "../../ir/build.js"; import { tryLowerExpression } from "./expressions/try-lower-expression.js"; import { lowerAddAbortListener } from "./builtin-calls.js"; import { requestGeneralStreamSurface } from "./general-streams.js"; @@ -931,6 +933,239 @@ function lowerDynListenerCall( }; } +/** The literal events typed code dispatches with a fixed tuple — it emits + * them or listens with typed parameters — whose names an untyped receiver + * may also use. 'error' and the meta events are excluded: the runtime fixes + * their tuples itself (scr_emitter_on_flex / scr_emitter_emit_flex). */ +function bridgedEvents(lowerer: Lowerer): { name: string; tuple: IrType[] }[] { + const table = emitterEvents(lowerer); + const events: { name: string; tuple: IrType[] }[] = []; + for (const [key, sig] of table) { + if (key === "error" || key === ERROR_MONITOR_EVENT || META_EVENTS.has(key)) continue; + if (sig.conflict !== null || (!sig.fromEmit && sig.typedListener !== true)) continue; + if (flexibleJsEvent(key, sig)) continue; + if (computedPatterns(lowerer).some((pattern) => patternMatchesName(pattern, key))) continue; + events.push({ name: eventKeyName(key), tuple: sig.tuple }); + } + return events; +} + +/** `emit(name, ...args)` through an untyped value on a (non-stream) + * emitter-rooted receiver, when typed code fixes the tuple of some event. + * Listeners registered as checked-dynamic functions receive the arguments + * as given; when the name is such an event and a listener with typed + * parameters is registered, the arguments also convert to the event's tuple + * (a mismatch throws the checked conversion's TypeError). Null when no event + * needs this, so the caller keeps the plain untyped emit. */ +export function lowerDynamicEmitterEmit( + lowerer: Lowerer, + receiver: IrExpr, + name: IrExpr, + payload: IrExpr[], + loc: SrcLoc, +): IrExpr | null { + const recvT = receiver.type; + if (recvT.kind !== "object") return null; + const k = payload.length; + const getRecord = (id: string) => lowerer.shapes.get(id); + const getUnion = (id: string) => lowerer.unions.get(id); + const checkable = (t: IrType): boolean => + t.kind === "dyn" || + isDynTypedRefType(t) || + isUnitType(t) || + (t.kind === "union" + ? (getUnion(t.unionId)?.arms.every(checkable) ?? false) + : canDynCheckTo(t, getRecord, getUnion)); + const converted = (tuple: IrType[]): IrExpr[] | null => { + if (!tuple.every(checkable)) return null; + const out: IrExpr[] = []; + for (let i = 0; i < tuple.length; i++) { + const arg = i < k ? varRef(`a.${i}`, DYN, loc) : dynUndefinedExpr(loc); + const value = lowerer.coerceToExpected(arg, tuple[i]!); + if (!typeEquals(value.type, tuple[i]!)) return null; + out.push(value); + } + return out; + }; + const events = bridgedEvents(lowerer) + .map((event) => ({ ...event, args: converted(event.tuple) })) + .filter((event) => event.args !== null); + if (events.length === 0) return null; + const key = `emitter.dynEmit:${typeKey(recvT)}:${k}`; + let helper = lowerer.arrHofHelpers.get(key); + if (!helper) { + helper = `%emitter.dynEmit.${lowerer.arrHofHelpers.size}`; + lowerer.arrHofHelpers.set(key, helper); + const params: IrParam[] = [ + { localId: "r.0", name: "r", type: recvT }, + { localId: "n.0", name: "n", type: STRING }, + ...payload.map((_, i) => ({ localId: `a.${i}`, name: `a${i}`, type: DYN })), + ]; + const r = varRef("r.0", recvT, loc); + const n = varRef("n.0", STRING, loc); + const dynArgs = payload.map((_, i) => varRef(`a.${i}`, DYN, loc)); + const body: IrStmt[] = events.map((event): IrStmt => ({ + kind: "if", + cond: { + kind: "strEq", + negated: false, + left: n, + right: strLit(event.name, loc), + type: BOOL, + loc, + }, + then: [ + { + kind: "if", + cond: { kind: "libCall", fn: "emitter.hasTyped", args: [r, n], type: BOOL, loc }, + then: [ + { + kind: "return", + value: { + kind: "libCall", + fn: "emitter.emitMixed", + args: [ + r, + n, + { kind: "numLit", value: k, type: F64, loc }, + ...dynArgs, + ...event.args!, + ], + type: BOOL, + loc, + }, + loc, + }, + ], + else_: null, + loc, + }, + ], + else_: null, + loc, + })); + body.push({ + kind: "return", + value: { kind: "libCall", fn: "emitter.emitAny", args: [r, n, ...dynArgs], type: BOOL, loc }, + loc, + }); + lowerer.liftedFns.push({ + name: helper, + params, + returnType: BOOL, + locals: params.map((p) => ({ id: p.localId, name: p.name, type: p.type, mutable: false })), + body, + loc, + }); + } + return { kind: "call", callee: helper, args: [receiver, name, ...payload], type: BOOL, loc }; +} + +/** on/once/prepend*(name, fn) through an untyped value on a (non-stream) + * emitter-rooted receiver, when typed code fixes the tuple of some event. + * Under such an event's name the function registers with a typed adapter + * for typed emits as well as its checked-dynamic thunk for untyped ones; + * other names register the thunk alone. Null when no event needs this. */ +export function lowerDynamicEmitterOn( + lowerer: Lowerer, + receiver: IrExpr, + name: IrExpr, + cb: IrExpr, + once: boolean, + prepend: boolean, + loc: SrcLoc, +): IrExpr | null { + const recvT = receiver.type; + if (recvT.kind !== "object") return null; + const getRecord = (id: string) => lowerer.shapes.get(id); + const getUnion = (id: string) => lowerer.unions.get(id); + const events = bridgedEvents(lowerer).filter((event) => + event.tuple.every((t) => t.kind === "dyn" || canConvertToDyn(t, getRecord, getUnion)), + ); + if (events.length === 0) return null; + const key = `emitter.dynOn:${typeKey(recvT)}`; + let helper = lowerer.arrHofHelpers.get(key); + if (!helper) { + helper = `%emitter.dynOn.${lowerer.arrHofHelpers.size}`; + lowerer.arrHofHelpers.set(key, helper); + const params: IrParam[] = [ + { localId: "r.0", name: "r", type: recvT }, + { localId: "n.0", name: "n", type: STRING }, + { localId: "cb.0", name: "cb", type: DYN }, + { localId: "o.0", name: "o", type: BOOL }, + { localId: "p.0", name: "p", type: BOOL }, + ]; + const locals: IrLocal[] = params.map((p) => ({ + id: p.localId, + name: p.name, + type: p.type, + mutable: false, + })); + const r = varRef("r.0", recvT, loc); + const n = varRef("n.0", STRING, loc); + const fn = varRef("cb.0", DYN, loc); + const o = varRef("o.0", BOOL, loc); + const p = varRef("p.0", BOOL, loc); + const body: IrStmt[] = [ + { + kind: "exprStmt", + expr: { kind: "libCall", fn: "emitter.checkListener", args: [fn], type: VOID, loc }, + loc, + }, + ]; + events.forEach((event, i) => { + const adapterT: IrType = { kind: "func", params: event.tuple, ret: VOID }; + const adapter = `a.${i}`; + locals.push({ id: adapter, name: "a", type: adapterT, mutable: false }); + body.push({ + kind: "if", + cond: { + kind: "strEq", + negated: false, + left: n, + right: strLit(event.name, loc), + type: BOOL, + loc, + }, + then: [ + { + kind: "varDecl", + localId: adapter, + init: { kind: "dynCheck", value: fn, type: adapterT, loc }, + loc, + }, + { + kind: "return", + value: { + kind: "libCall", + fn: "emitter.onDyn", + args: [r, n, fn, varRef(adapter, adapterT, loc), o, p], + type: recvT, + loc, + }, + loc, + }, + ], + else_: null, + loc, + }); + }); + body.push({ + kind: "return", + value: { kind: "libCall", fn: "emitter.onAny", args: [r, n, fn, o, p], type: recvT, loc }, + loc, + }); + lowerer.liftedFns.push({ name: helper, params, returnType: recvT, locals, body, loc }); + } + return { + kind: "call", + callee: helper, + args: [receiver, name, cb, boolLit(once, loc), boolLit(prepend, loc)], + type: recvT, + loc, + }; +} + /** A listener argument: lowered, checked void-returning, and its * parameters checked as a typeEquals PREFIX of the event tuple. */ function lowerListenerArg( diff --git a/packages/compiler/src/ir/builtin-effects.ts b/packages/compiler/src/ir/builtin-effects.ts index d4e96a35a..8696e2139 100644 --- a/packages/compiler/src/ir/builtin-effects.ts +++ b/packages/compiler/src/ir/builtin-effects.ts @@ -232,12 +232,16 @@ export const MAY_THROW_LIB_FNS: ReadonlySet = new Set([ // 'error' has no listener, Node's unhandled-'error' contract. "emitter.emit", "emitter.emitFlex", + "emitter.emitAny", + "emitter.emitMixed", + "emitter.listenersDyn", "emitter.emitError", "emitter.on", "emitter.off", "emitter.checkListener", "emitter.onDyn", "emitter.onFlex", + "emitter.onAny", "emitter.offDyn", "emitter.onData", "emitter.onDataDyn", diff --git a/packages/compiler/src/ir/builtin-signatures.ts b/packages/compiler/src/ir/builtin-signatures.ts index 1ffae7536..4dc7b69e0 100644 --- a/packages/compiler/src/ir/builtin-signatures.ts +++ b/packages/compiler/src/ir/builtin-signatures.ts @@ -3716,16 +3716,31 @@ export const LIB_FN_SIGS = defineLibFnSignatures({ "emitter.checkListener": { argTypes: [DYN], result: VOID }, "emitter.onDyn": { argTypes: [null, STRING, DYN, null, BOOL, BOOL], result: VOID }, "emitter.onFlex": { argTypes: [null, STRING, DYN, BOOL, BOOL], result: VOID }, + /** on/once/prepend* and emit through an untyped value (the class + * dispatch of compiled emitters): onFlex/emitFlex plus runtime typed + * adapters under 'error' and the meta events, and Node's unhandled-'error' + * contract. emitAny is VARIADIC like emitFlex. */ + "emitter.onAny": { argTypes: [null, STRING, DYN, BOOL, BOOL], result: VOID }, "emitter.offDyn": { argTypes: [null, STRING, DYN], result: VOID }, "emitter.removeAll": { argTypes: [null, STRING, BOOL], result: VOID }, "emitter.emit": { argTypes: [null, STRING], result: BOOL }, "emitter.emitFlex": { argTypes: [null, STRING], result: BOOL }, + "emitter.emitAny": { argTypes: [null, STRING], result: BOOL }, + /** An untyped emit of an event with a fixed tuple: (recv, name, k, then + * k checked-dynamic arguments as given, then the event's converted + * tuple) — VARIADIC like emitter.emit. hasTyped answers whether a + * listener of the name needs the converted tuple. */ + "emitter.emitMixed": { argTypes: [null, STRING, F64], result: BOOL }, + "emitter.hasTyped": { argTypes: [null, STRING], result: BOOL }, "emitter.emitError": { argTypes: [null, STRING, null], result: BOOL }, "emitter.count": { argTypes: [null, STRING], result: F64 }, "emitter.countFn": { argTypes: [null, STRING, null], result: F64 }, "emitter.countDyn": { argTypes: [null, STRING, DYN], result: F64 }, "emitter.names": { argTypes: [null], result: VOID }, "emitter.listeners": { argTypes: [null, STRING], result: VOID }, + /** listeners()/rawListeners() through an untyped value: a checked- + * dynamic array (throws when a listener has typed parameters). */ + "emitter.listenersDyn": { argTypes: [null, STRING], result: DYN }, /** Stream-'data' registration twins of emitter.on/onDyn (same args): * chosen at registration sites whose receiver is stream-rooted, so the * backend emits DATA thunks — the runtime's 'data' emission carries diff --git a/packages/compiler/src/ir/validate.ts b/packages/compiler/src/ir/validate.ts index 2a2ddbd2f..3ec68f28c 100644 --- a/packages/compiler/src/ir/validate.ts +++ b/packages/compiler/src/ir/validate.ts @@ -4061,6 +4061,8 @@ function validateFunction( const variadic = e.fn === "emitter.emit" || e.fn === "emitter.emitFlex" || + e.fn === "emitter.emitAny" || + e.fn === "emitter.emitMixed" || e.fn === "readable.new" || e.fn === "writable.new" || e.fn === "duplex.new" || @@ -5828,6 +5830,7 @@ function validateFunction( e.fn === "emitter.off" || e.fn === "emitter.onDyn" || e.fn === "emitter.onFlex" || + e.fn === "emitter.onAny" || e.fn === "stream.onDyn" || e.fn === "emitter.offDyn" || e.fn === "emitter.onData" || @@ -5882,14 +5885,27 @@ function validateFunction( if (!ok) err(`libCall emitter.listeners must return a func array`, e.loc); return true; } - if (e.fn === "emitter.emitFlex") { + if (e.fn === "emitter.emitFlex" || e.fn === "emitter.emitAny") { if (e.args.slice(2).some((arg) => arg.type.kind !== "dyn")) { - err("libCall emitter.emitFlex payloads must be checked-dynamic", e.loc); + err(`libCall ${e.fn} payloads must be checked-dynamic`, e.loc); + } + } + if (e.fn === "emitter.emitMixed") { + const count = e.args[2]; + const k = count?.kind === "numLit" ? count.value : -1; + if (!Number.isInteger(k) || k < 0 || e.args.length < 3 + k) { + err("libCall emitter.emitMixed needs a literal argument count", e.loc); + } else if (e.args.slice(3, 3 + k).some((arg) => arg.type.kind !== "dyn")) { + err("libCall emitter.emitMixed untyped arguments must be checked-dynamic", e.loc); } } if ( e.fn === "emitter.emit" || e.fn === "emitter.emitFlex" || + e.fn === "emitter.emitAny" || + e.fn === "emitter.emitMixed" || + e.fn === "emitter.hasTyped" || + e.fn === "emitter.listenersDyn" || e.fn === "emitter.count" || e.fn === "emitter.getMax" || e.fn === "emitter.ctor" || diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6580-emitter-typed-untyped-listeners.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6580-emitter-typed-untyped-listeners.ts.json new file mode 100644 index 000000000..347deebd8 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6580-emitter-typed-untyped-listeners.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6580-emitter-typed-untyped-listeners.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6581-emitter-untyped-error-and-meta.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6581-emitter-untyped-error-and-meta.ts.json new file mode 100644 index 000000000..a8252c615 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6581-emitter-untyped-error-and-meta.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6581-emitter-untyped-error-and-meta.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6582-emitter-class-expression-escapes.cjs.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6582-emitter-class-expression-escapes.cjs.json new file mode 100644 index 000000000..9cc32b856 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6582-emitter-class-expression-escapes.cjs.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6582-emitter-class-expression-escapes.cjs" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6583-emitter-untyped-surface.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6583-emitter-untyped-surface.ts.json new file mode 100644 index 000000000..b570e5d43 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6583-emitter-untyped-surface.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6583-emitter-untyped-surface.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6584-emitter-class-body-untyped-listeners.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6584-emitter-class-body-untyped-listeners.ts.json new file mode 100644 index 000000000..45c0a95ba --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6584-emitter-class-body-untyped-listeners.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6584-emitter-class-body-untyped-listeners.ts" + ], + "diags": [] +} diff --git a/packages/runtime/src/scr_events_emitter.c b/packages/runtime/src/scr_events_emitter.c index 4ac7abdf1..c11b7897d 100644 --- a/packages/runtime/src/scr_events_emitter.c +++ b/packages/runtime/src/scr_events_emitter.c @@ -27,8 +27,10 @@ * site's writes. Internal (meta-event) emits reuse the same variadic * entry, passing the event NAME as the one argument — meta listeners are * fenced to arity <= 1, so no adapter ever reads the listener argument - * Node would pass second. Computed names disjoint from internal events use a - * checked-dynamic callback and an exact-arity argument vector instead. + * Node would pass second. Listeners registered as checked-dynamic functions + * keep their checked-dynamic thunk, which untyped emits call with the exact + * argument vector; an entry can hold both forms, so typed and untyped + * registrations and emits of one name share one list. * * The special 'error' event routes through scr_emitter_emit_error: with no * 'error' listener the Error payload THROWS through the exception cell @@ -70,11 +72,19 @@ typedef struct ScrEeEntry { * listeners()/rawListeners() answer it — Node's identity semantics. */ ScrClosure *orig; ScrEeInvoke inv; - /* Computed-name registration: the original closure and its checked- - * dynamic thunk, called with the exact argument count. Holding the - * closure directly keeps its capture edges visible to cycle collection. */ + /* Checked-dynamic registration (computed names, and every listener + * registered as a dynamic function value): the original closure and its + * checked-dynamic thunk, called with the exact argument count of an + * untyped emit. Holding the closure directly keeps its capture edges + * visible to cycle collection. An entry may carry both forms: a typed + * emit calls `inv`, an untyped one calls the thunk with the arguments as + * given. The descriptor rebuilds the function value for listeners() and + * the meta events. */ ScrClosure *flex_cb; ScrDynThunk flex_invoke; + const char *flex_sig; + const char *flex_name; + uint32_t flex_arity; bool once; bool fired; /* a once entry that already ran (snapshot skip rule) */ uint32_t refs; @@ -355,6 +365,8 @@ void scr_emitter_release_v(void *em) { scr_emitter_release(em); } /* ── the meta events ('newListener' / 'removeListener') ──────────────── */ static bool scr_ee_emit_va(ScrEmitter *em, ScrStr *name, ...); +static bool scr_ee_emit_mixed_va(ScrEmitter *em, ScrStr *name, ScrDyn *const *args, size_t argc, + int typed, ...); static void scr_ee_symbol_name_fence(const char *what) { char message[160]; @@ -363,10 +375,62 @@ static void scr_ee_symbol_name_fence(const char *what) { scr_throw_error(SCR_ERR_ERROR, scr_str_new(message, strlen(message))); } -/* Fires a meta event with the affected event's NAME as the argument - * (listeners are fenced to arity <= 1, so the listener-function argument - * Node passes second is never read). */ -static void scr_ee_emit_meta(ScrEmitter *em, const char *meta, ScrStr *name) { +/* Whether any entry of the bucket was registered as a checked-dynamic + * function (it can take the untyped argument vector). */ +static bool scr_ee_bucket_has_flex(const ScrEeBucket *b) { + for (size_t i = 0; b && i < b->n; i++) { + if (b->ls[i]->flex_invoke) return true; + } + return false; +} + +/* The checked-dynamic function value of a dynamically registered entry + * (+1), or NULL for a typed listener. Identity lives in the closure, so the + * rebuilt box compares equal to the value that was registered. */ +static ScrDyn *scr_ee_entry_dyn(const ScrEeEntry *e) { + if (!e->flex_invoke) return NULL; + return scr_dyn_new_func(scr_closure_retain(e->flex_cb), e->flex_invoke, e->flex_arity, + e->flex_sig, e->flex_name); +} + +/* A meta event with a listener registered as a checked-dynamic function: + * the untyped argument vector — the affected event's name and, when it was + * registered dynamically, the listener (`fn`, borrowed, may be NULL) — + * beside the typed one-name tuple. False when no such listener exists (the + * caller emits the typed tuple alone). */ +static bool scr_ee_dyn_emit_meta(ScrEmitter *em, ScrStr *meta, ScrStr *name, const ScrDyn *fn) { + if (!scr_ee_bucket_has_flex(scr_ee_bucket_of(em->reg, meta))) return false; + ScrDyn *args[2] = { scr_dyn_new_str(name), (ScrDyn *)fn }; + scr_ee_emit_mixed_va(em, meta, args, fn ? 2 : 1, 1, name); + scr_dyn_release(args[0]); + return true; +} + +/* The function value of a listener about to be removed, for checked- + * dynamic 'removeListener' listeners (+1), or NULL. */ +static ScrDyn *scr_ee_dyn_removed(ScrEmitter *em, const ScrEeEntry *e) { + if (!scr_ee_bucket_has_flex(scr_ee_bucket_find(em->reg, "removeListener", 14))) return NULL; + return scr_ee_entry_dyn(e); +} + +/* The checked-dynamic half of the registry, installed by the first + * registration through an untyped value (scr_emitter_on_any): only such + * listeners take the meta events' untyped argument vector, so programs + * without untyped emitter calls never reach, or link, this half. */ +struct ScrEeDynOps { + bool (*emit_meta)(ScrEmitter *em, ScrStr *meta, ScrStr *name, const ScrDyn *fn); + ScrDyn *(*removed)(ScrEmitter *em, const struct ScrEeEntry *e); +}; +static const ScrEeDynOps scr_ee_dyn_ops = { &scr_ee_dyn_emit_meta, &scr_ee_dyn_removed }; +/* External linkage on purpose: the optimizer must not prove the hook's + * value from this unit's stores and inline the dynamic half back in. */ +SCR_TL const ScrEeDynOps *scr_ee_dyn = NULL; + +/* Fires a meta event with the affected event's NAME as the argument. + * Typed meta listeners are fenced to arity <= 1; listeners registered as + * checked-dynamic functions also receive the listener Node passes second + * when it was itself registered dynamically (`fn`, borrowed, may be NULL). */ +static void scr_ee_emit_meta(ScrEmitter *em, const char *meta, ScrStr *name, const ScrDyn *fn) { if (!scr_ee_has(em->reg, meta)) return; /* Node passes the symbol itself, which the string tuple cannot carry. */ if (scr_ee_is_error_monitor(name)) { @@ -374,7 +438,7 @@ static void scr_ee_emit_meta(ScrEmitter *em, const char *meta, ScrStr *name) { return; } ScrStr *m = scr_str_new(meta, strlen(meta)); - scr_ee_emit_va(em, m, name); + if (!scr_ee_dyn || !scr_ee_dyn->emit_meta(em, m, name, fn)) scr_ee_emit_va(em, m, name); scr_str_release(m); } @@ -435,19 +499,23 @@ static void scr_ee_warn_maybe(ScrEmitter *em, ScrEeBucket *b) { * try). */ static ScrEmitter *scr_ee_add(ScrEmitter *em, ScrStr *name, ScrClosure *cb /*moves*/, ScrClosure *orig /*moves; may be NULL*/, - ScrEeInvoke inv, ScrClosure *flex_cb /*moves; may be NULL*/, - ScrDynThunk flex_invoke, + ScrEeInvoke inv, const ScrDyn *dyn /*borrowed; may be NULL*/, bool once, bool prepend) { scr_ee_reg_ensure(em); - scr_ee_emit_meta(em, "newListener", name); + scr_ee_emit_meta(em, "newListener", name, dyn); ScrEeBucket *b = scr_ee_bucket_ensure(em->reg, name); ScrEeEntry *e = calloc(1, sizeof *e); if (!e) scr_ee_oom(); e->cb = cb; e->orig = orig; e->inv = inv; - e->flex_cb = flex_cb; - e->flex_invoke = flex_invoke; + if (dyn) { + e->flex_cb = scr_closure_retain(dyn->v.fn.clo); + e->flex_invoke = dyn->v.fn.thunk; + e->flex_sig = dyn->v.fn.sig; + e->flex_name = dyn->v.fn.name; + e->flex_arity = dyn->v.fn.arity; + } e->once = once; e->refs = 1; if (b->n == b->cap) { @@ -470,7 +538,7 @@ static ScrEmitter *scr_ee_add(ScrEmitter *em, ScrStr *name, ScrClosure *cb /*mov ScrEmitter *scr_emitter_on(ScrEmitter *em, ScrStr *name, ScrClosure *cb /*moves*/, ScrEeInvoke inv, bool once, bool prepend) { - return scr_ee_add(em, name, cb, NULL, inv, NULL, NULL, once, prepend); + return scr_ee_add(em, name, cb, NULL, inv, NULL, once, prepend); } /* events.once(emitter, name): a pending promise of the event's arguments @@ -553,7 +621,7 @@ ScrPromise *scr_events_once(ScrEmitter *em, ScrStr *name, ScrClosure *boxer, voi * listener it announced is never added (the event's listener stays when * only the 'error' announcement throws). */ ScrCaught *thrown = NULL; - scr_emitter_release(scr_ee_add(em, name, st->event_cb, NULL, inv, NULL, NULL, true, false)); + scr_emitter_release(scr_ee_add(em, name, st->event_cb, NULL, inv, NULL, true, false)); if (scr_exc_pending()) { thrown = scr_exc_take(); scr_ee_unadd(em, name, st->event_cb); @@ -561,7 +629,7 @@ ScrPromise *scr_events_once(ScrEmitter *em, ScrStr *name, ScrClosure *boxer, voi st->event_cb = st->error_cb = NULL; } else if (st->error_cb != NULL) { ScrStr *error = scr_str_new("error", 5); - scr_emitter_release(scr_ee_add(em, error, st->error_cb, NULL, err_inv, NULL, NULL, true, false)); + scr_emitter_release(scr_ee_add(em, error, st->error_cb, NULL, err_inv, NULL, true, false)); if (scr_exc_pending()) { thrown = scr_exc_take(); scr_ee_unadd(em, error, st->error_cb); @@ -613,23 +681,87 @@ void scr_events_once_fire(ScrClosure *cb, ScrArr *args) { /* The dyn-adapted registration (JS-lane checked-dynamic listeners): the * ADAPTER closure (compiler-built — boxes the event tuple to dyn and - * calls the original through the checked-dynamic machinery) is what emit - * invokes; the ORIGINAL function (the dyn box's underlying closure) is - * the entry's identity. cb must be a SCR_DYN_FUNC (the emitted - * scr_emitter_check_listener ran first); adapter MOVES in, cb is - * borrowed. */ + * calls the original through the checked-dynamic machinery) is what a + * typed emit invokes; the ORIGINAL function (the dyn box's underlying + * closure) is the entry's identity, and its checked-dynamic thunk is what + * an untyped emit of the same name invokes, with every argument as given. + * cb must be a SCR_DYN_FUNC (the emitted scr_emitter_check_listener ran + * first); adapter MOVES in, cb is borrowed. */ ScrEmitter *scr_emitter_on_dyn(ScrEmitter *em, ScrStr *name, const ScrDyn *cb, ScrClosure *adapter /*moves*/, ScrEeInvoke inv, bool once, bool prepend) { - return scr_ee_add(em, name, adapter, scr_closure_retain(cb->v.fn.clo), inv, NULL, NULL, once, - prepend); + return scr_ee_add(em, name, adapter, scr_closure_retain(cb->v.fn.clo), inv, + cb->kind == SCR_DYN_FUNC ? cb : NULL, once, prepend); } +/* The typed adapters of a checked-dynamic listener on the events whose + * tuples the runtime itself fixes: 'error' (one Error) and the meta + * events (one name string). caps[0] holds the dynamic function in a traced + * box, so a listener capturing its emitter stays collectable. */ +static void scr_ee_dyn_fire(ScrClosure *self, ScrDyn *arg /*moves*/) { + ScrDyn *fn = scr_box_get_ref(self->caps[0]); /* +1 */ + ScrDyn *args[1] = { arg }; + scr_dyn_release(scr_dyn_call(fn, args, 1, "listener")); + scr_dyn_release(arg); + scr_dyn_release(fn); +} + +static void scr_ee_dyn_error_fire(ScrClosure *self, ScrError *err) { + scr_ee_dyn_fire(self, scr_dyn_from_error(err)); +} + +static void scr_ee_dyn_name_fire(ScrClosure *self, ScrStr *name) { + scr_ee_dyn_fire(self, scr_dyn_new_str(name)); +} + +static ScrClosure *scr_ee_dyn_adapter(const ScrDyn *cb, void *fire) { + ScrClosure *clo = scr_closure_new(fire, 1); + ScrBox *box = scr_box_new_obj(&scr_dyn_retain_v, &scr_dyn_release_v, &scr_dyn_trace_v); + scr_box_set_ref(box, scr_dyn_retain((ScrDyn *)cb)); + clo->caps[0] = box; + return clo; +} + +static bool scr_ee_is_name(const ScrStr *name, const char *literal) { + size_t len = strlen(literal); + return name->len == len && memcmp(name->data, literal, len) == 0; +} + +/* A listener registered as a checked-dynamic function (computed event + * names, and runtime-internal registries such as the worker events, whose + * emits are untyped): its thunk alone. cb must be a function value (the + * caller ran the listener check). */ ScrEmitter *scr_emitter_on_flex(ScrEmitter *em, ScrStr *name, const ScrDyn *cb, bool once, bool prepend) { - if (cb->kind != SCR_DYN_FUNC) scr_trap("scriptc: computed event listener is not a function\n"); - return scr_ee_add(em, name, NULL, NULL, NULL, scr_closure_retain(cb->v.fn.clo), - cb->v.fn.thunk, once, prepend); + if (cb->kind != SCR_DYN_FUNC) { + scr_dyn_check_listener(cb, "listener"); + return scr_emitter_retain(em); + } + return scr_ee_add(em, name, NULL, NULL, NULL, cb, once, prepend); +} + +/* on/once/prepend* through an untyped value, under a name the program never + * fixes a tuple for: the thunk, and under 'error' and the meta events, whose + * tuples typed code and the runtime emit with, a runtime-built typed adapter + * too. Installs the registry's dynamic half. */ +ScrEmitter *scr_emitter_on_any(ScrEmitter *em, ScrStr *name, const ScrDyn *cb, + bool once, bool prepend) { + if (cb->kind != SCR_DYN_FUNC) { + scr_dyn_check_listener(cb, "listener"); + return scr_emitter_retain(em); + } + scr_ee_dyn = &scr_ee_dyn_ops; + void *fire = NULL; + if (!scr_ee_is_error_monitor(name)) { + if (scr_ee_is_name(name, "error")) fire = (void *)&scr_ee_dyn_error_fire; + else if (scr_ee_is_name(name, "newListener") || scr_ee_is_name(name, "removeListener")) + fire = (void *)&scr_ee_dyn_name_fire; + } + if (fire) { + return scr_ee_add(em, name, scr_ee_dyn_adapter(cb, fire), + scr_closure_retain(cb->v.fn.clo), &scr_ee_inv_fixed1, cb, once, prepend); + } + return scr_ee_add(em, name, NULL, NULL, NULL, cb, once, prepend); } /* The adapter-mediated registration (the LLVM backend): adapter is what @@ -638,7 +770,7 @@ ScrEmitter *scr_emitter_on_flex(ScrEmitter *em, ScrStr *name, const ScrDyn *cb, ScrEmitter *scr_emitter_on_via(ScrEmitter *em, ScrStr *name, ScrClosure *orig /*moves*/, ScrClosure *adapter /*moves*/, ScrEeInvoke inv, bool once, bool prepend) { - return scr_ee_add(em, name, adapter, orig, inv, NULL, NULL, once, prepend); + return scr_ee_add(em, name, adapter, orig, inv, NULL, once, prepend); } /* ── fixed-arity invoke shims (scr_runtime.h's contract) ────────────── @@ -702,8 +834,12 @@ static void scr_ee_remove_at_quiet(ScrEmitter *em, ScrEeBucket *b, size_t i) { static void scr_ee_remove_at(ScrEmitter *em, ScrEeBucket *b, size_t i) { ScrStr *name = scr_str_retain(b->name); + /* The removed listener as a function value, for checked-dynamic + * 'removeListener' listeners (Node's second argument). */ + ScrDyn *fn = scr_ee_dyn ? scr_ee_dyn->removed(em, b->ls[i]) : NULL; scr_ee_remove_at_quiet(em, b, i); - scr_ee_emit_meta(em, "removeListener", name); + scr_ee_emit_meta(em, "removeListener", name, fn); + if (fn) scr_dyn_release(fn); scr_str_release(name); } @@ -858,14 +994,25 @@ ScrEmitter *scr_emitter_remove_all(ScrEmitter *em, ScrStr *name, bool all) { /* ── emit ─────────────────────────────────────────────────────────────── */ -/* One dispatch pass over a SNAPSHOT of the bucket: entries invoke in list - * order through their emitted adapters, each with a fresh va_start over - * the emit site's typed tail. A once entry leaves the live list (firing - * 'removeListener', Node's wrapper calls removeListener) BEFORE its - * callback runs, and never runs twice even when a nested emit's snapshot - * still holds it (the shared fired flag — Node's wrapper.fired). A - * listener throw stops the pass (the exception stays pending for the - * caller). Returns whether the bucket had listeners. */ +/* The checked-dynamic argument vector of an untyped emit without + * arguments: flex_args is non-NULL exactly when the emit carries one. */ +static ScrDyn *const scr_ee_no_args[1] = { NULL }; + +/* A typed emit site passes its event's tuple as C variadics (ap_template); + * an untyped emit passes the checked-dynamic arguments as given + * (flex_args); an untyped emit of an event typed code also uses passes both + * (the compiler converted the arguments to the event's tuple for listeners + * with typed parameters). + * + * One dispatch pass over a SNAPSHOT of the bucket: entries invoke in list + * order, each listener registered as a checked-dynamic function through its + * thunk when the emit carries untyped arguments, otherwise through its + * emitted adapter with a fresh va_copy of the typed tail. A once entry + * leaves the live list (firing 'removeListener', Node's wrapper calls + * removeListener) BEFORE its callback runs, and never runs twice even when + * a nested emit's snapshot still holds it (the shared fired flag — Node's + * wrapper.fired). A listener throw stops the pass (the exception stays + * pending for the caller). Returns whether the bucket had listeners. */ static bool scr_ee_emit_core(ScrEmitter *em, ScrStr *name, va_list *ap_template, ScrDyn *const *flex_args, size_t flex_argc); @@ -877,6 +1024,26 @@ static void scr_ee_emit_error_monitor(ScrEmitter *em, va_list *ap_template, scr_ee_emit_core(em, SCR_EE_ERROR_MONITOR, ap_template, flex_args, flex_argc); } +/* A listener this emit has no argument form for: an untyped emit whose + * arguments have no conversion to a typed listener's parameters, or a typed + * emit reaching a dynamic listener without a typed adapter. Throws (the + * pass stops like a listener throw); never traps. */ +SCR_COLD static void scr_ee_listener_mismatch(const ScrStr *name, bool typed_emit) { + ScrJsonBuf text; + scr_jb_init(&text); + if (typed_emit) { + scr_jb_puts(&text, "emitting '"); + scr_jb_write(&text, name->data, name->len); + scr_jb_puts(&text, "' with typed arguments to a listener registered through an untyped value is not supported yet"); + } else { + scr_jb_puts(&text, "The '"); + scr_jb_write(&text, name->data, name->len); + scr_jb_puts(&text, "' listener has typed parameters the arguments of this untyped emit do not convert to"); + } + ScrStr *message = scr_jb_finish(&text); + scr_throw_error(typed_emit ? SCR_ERR_ERROR : SCR_ERR_TYPE, message); +} + static bool scr_ee_emit_core(ScrEmitter *em, ScrStr *name, va_list *ap_template, ScrDyn *const *flex_args, size_t flex_argc) { if (name->len == 5 && memcmp(name->data, "error", 5) == 0) { @@ -921,16 +1088,16 @@ static bool scr_ee_emit_core(ScrEmitter *em, ScrStr *name, va_list *ap_template, continue; } } - if (ap_template) { - if (!e->inv) scr_trap("scriptc: computed event collided with a fixed event\n"); + if (flex_args && e->flex_invoke) { + ScrDyn *answer = e->flex_invoke(e->flex_cb, flex_args, flex_argc); + scr_dyn_release(answer); /* EventEmitter ignores callback results. */ + } else if (ap_template && e->inv) { va_list ap; va_copy(ap, *ap_template); e->inv(e->cb, ap); va_end(ap); } else { - if (!e->flex_invoke) scr_trap("scriptc: computed event collided with a fixed event\n"); - ScrDyn *answer = e->flex_invoke(e->flex_cb, flex_args, flex_argc); - scr_dyn_release(answer); /* EventEmitter ignores callback results. */ + scr_ee_listener_mismatch(name, ap_template != NULL); } scr_ee_entry_unref(e); } @@ -946,6 +1113,17 @@ static bool scr_ee_emit_va(ScrEmitter *em, ScrStr *name, ...) { return had; } +/* An untyped emit with the typed tuple too when `typed` (the variadics), + * or the untyped arguments alone. */ +static bool scr_ee_emit_mixed_va(ScrEmitter *em, ScrStr *name, ScrDyn *const *args, size_t argc, + int typed, ...) { + va_list ap; + va_start(ap, typed); + bool had = scr_ee_emit_core(em, name, typed ? &ap : NULL, args ? args : scr_ee_no_args, argc); + va_end(ap); + return had; +} + /* emit(name, ...tuple) — the emitted call site passes the event's unified * tuple as typed C variadics (doubles, pointers, ints for bools); every * argument is BORROWED (adapters retain the +1 each callee owns). May @@ -958,11 +1136,126 @@ bool scr_emitter_emit(ScrEmitter *em, ScrStr *name, ...) { return had; } -bool scr_emitter_emit_flex(ScrEmitter *em, ScrStr *name, ScrDyn *const *args, size_t argc) { - if (name->len == 5 && memcmp(name->data, "error", 5) == 0) { - scr_trap("scriptc: computed event collided with the error event\n"); +/* An untyped emit of an event typed code also emits or listens to: the + * arguments as given (borrowed) for listeners registered as checked-dynamic + * functions, and the compiler-converted tuple (the variadics, scr_emitter_emit's + * form) for typed listeners. */ +bool scr_emitter_emit_mixed(ScrEmitter *em, ScrStr *name, ScrDyn *const *args, size_t argc, ...) { + va_list ap; + va_start(ap, argc); + bool had = scr_ee_emit_core(em, name, &ap, args ? args : scr_ee_no_args, argc); + va_end(ap); + return had; +} + +/* Whether an untyped emit of `name` reaches a listener without a checked- + * dynamic thunk (one registered with typed parameters), so the arguments + * must convert to the event's tuple first. Includes errorMonitor for + * 'error'. */ +bool scr_emitter_has_typed(ScrEmitter *em, ScrStr *name) { + ScrEeBucket *b = em->reg ? scr_ee_bucket_of(em->reg, name) : NULL; + for (size_t i = 0; b && i < b->n; i++) { + if (!b->ls[i]->flex_invoke) return true; } - return scr_ee_emit_core(em, name, NULL, args, argc); + if (!scr_ee_is_error_monitor(name) && scr_ee_is_name(name, "error")) + return scr_emitter_has_typed(em, SCR_EE_ERROR_MONITOR); + return false; +} + +/* Node's ERR_UNHANDLED_ERROR for a non-Error payload: the message quotes + * util.inspect of the payload. This unit renders the primitive payloads + * inspect prints plainly; others report that they are not supported. */ +static void scr_ee_unhandled_error(const ScrDyn *er) { + ScrJsonBuf text; + scr_jb_init(&text); + scr_jb_puts(&text, "Unhandled error. ("); + bool shown = true; + switch (er->kind) { + case SCR_DYN_UNDEF: scr_jb_puts(&text, "undefined"); break; + case SCR_DYN_NULL: scr_jb_puts(&text, "null"); break; + case SCR_DYN_BOOL: scr_jb_puts(&text, er->v.b ? "true" : "false"); break; + case SCR_DYN_NUM: { + char num[32]; + if (er->v.num == 0 && 1 / er->v.num < 0) { + scr_jb_puts(&text, "-0"); + } else { + size_t len = scr_f64_to_str(er->v.num, num); + scr_jb_write(&text, num, len); + } + break; + } + case SCR_DYN_STR: { + const ScrStr *s = er->v.str; + for (size_t i = 0; i < s->len && shown; i++) { + unsigned char c = (unsigned char)s->data[i]; + if (c < 0x20 || c == 0x7f || c == '\'' || c == '\\') shown = false; + } + if (s->len > 10000) shown = false; + if (shown) { + scr_jb_putc(&text, '\''); + scr_jb_write(&text, s->data, s->len); + scr_jb_putc(&text, '\''); + } + break; + } + default: shown = false; break; + } + ScrStr *message = scr_jb_finish(&text); + if (shown) { + ScrJsonBuf full; + scr_jb_init(&full); + scr_jb_write(&full, message->data, message->len); + scr_jb_putc(&full, ')'); + ScrStr *rendered = scr_jb_finish(&full); + scr_throw_error_msg_code(SCR_ERR_ERROR, rendered->data, rendered->len, "ERR_UNHANDLED_ERROR"); + scr_str_release(rendered); + } else { + const char *refusal = + "emit('error') through an untyped value with this non-Error payload and no 'error' listener is not supported yet"; + scr_throw_error(SCR_ERR_ERROR, scr_str_new(refusal, strlen(refusal))); + } + scr_str_release(message); +} + +/* emit('error', ...) through an untyped value: errorMonitor first; with no + * 'error' listener an Error payload throws itself and anything else throws + * ERR_UNHANDLED_ERROR, Node's contract; otherwise listeners receive the + * arguments as given, and typed listeners the payload as an Error. */ +static bool scr_ee_emit_error_dyn(ScrEmitter *em, ScrStr *name, ScrDyn *const *args, size_t argc) { + const ScrDyn *er = argc ? args[0] : scr_dyn_undefined(); + ScrError *err = scr_dyn_err_instanceof(er, SCR_ERR_ERROR) ? scr_errdyn_err_of(er) : NULL; + bool had = false; + if (!scr_ee_has(em->reg, "error")) { + scr_ee_emit_mixed_va(em, SCR_EE_ERROR_MONITOR, args, argc, err != NULL, err); + if (!scr_exc_pending()) { + if (err) scr_dyn_throw(scr_dyn_retain((ScrDyn *)er)); + else scr_ee_unhandled_error(er); + } + } else { + had = scr_ee_emit_mixed_va(em, name, args, argc, err != NULL, err); + } + scr_error_release(err); + return had; +} + +/* An untyped emit of a name typed code never fixes a tuple for (computed + * names, runtime-internal registries): the arguments as given. */ +bool scr_emitter_emit_flex(ScrEmitter *em, ScrStr *name, ScrDyn *const *args, size_t argc) { + return scr_ee_emit_core(em, name, NULL, args ? args : scr_ee_no_args, argc); +} + +/* emit(name, ...args) through an untyped value: the arguments as given. + * 'error' keeps Node's unhandled-error contract, and the meta events reach + * typed listeners with a string name. */ +bool scr_emitter_emit_any(ScrEmitter *em, ScrStr *name, ScrDyn *const *args, size_t argc) { + if (!scr_ee_is_error_monitor(name)) { + if (scr_ee_is_name(name, "error")) return scr_ee_emit_error_dyn(em, name, args, argc); + if (scr_ee_is_name(name, "newListener") || scr_ee_is_name(name, "removeListener")) { + ScrStr *s = argc && args[0]->kind == SCR_DYN_STR ? args[0]->v.str : NULL; + return scr_ee_emit_mixed_va(em, name, args, argc, s != NULL, s); + } + } + return scr_ee_emit_core(em, name, NULL, args ? args : scr_ee_no_args, argc); } /* Handle event registries use checked callbacks for every tuple, including @@ -1073,6 +1366,26 @@ ScrArr *scr_emitter_listeners(ScrEmitter *em, ScrStr *name) { return out; } +/* listeners(name) / rawListeners(name) through an untyped value: a +1 + * checked-dynamic array of the listener functions in list order. A + * listener registered with typed parameters has no checked-dynamic + * function value, so such a bucket reports that it is not supported. */ +ScrDyn *scr_emitter_listeners_dyn(ScrEmitter *em, ScrStr *name) { + ScrDyn *out = scr_dyn_new_arr(); + ScrEeBucket *b = em->reg ? scr_ee_bucket_of(em->reg, name) : NULL; + for (size_t i = 0; b && i < b->n; i++) { + ScrDyn *fn = scr_ee_entry_dyn(b->ls[i]); + if (!fn) { + scr_dyn_release(out); + const char *message = "listeners() through an untyped value of an event with a typed listener is not supported yet"; + scr_throw_error(SCR_ERR_ERROR, scr_str_new(message, strlen(message))); + return scr_dyn_new_arr(); + } + scr_dyn_arr_push(out, fn); + } + return out; +} + /* ── max listeners ────────────────────────────────────────────────────── */ /* The checked-dynamic setMaxListeners ladder: non-numbers throw Node's diff --git a/packages/runtime/src/scr_events_emitter.h b/packages/runtime/src/scr_events_emitter.h index 39a7f4b5b..b66de9e5a 100644 --- a/packages/runtime/src/scr_events_emitter.h +++ b/packages/runtime/src/scr_events_emitter.h @@ -95,12 +95,27 @@ void scr_emitter_check_listener(const ScrDyn *cb); ScrEmitter *scr_emitter_on_dyn(ScrEmitter *em, ScrStr *name, const ScrDyn *cb, ScrClosure *adapter /*moves*/, ScrEeInvoke inv, bool once, bool prepend); -/* Computed-name listeners retain the original closure and its checked- - * dynamic call thunk, preserving the number of arguments passed to emit. - * The frontend keeps these names disjoint from fixed-tuple event names. */ +/* Listeners registered as checked-dynamic functions retain the original + * closure and its checked-dynamic call thunk, preserving the number of + * arguments passed to an untyped emit. on_dyn entries also carry the typed + * adapter for the event's fixed tuple; on_flex entries are for names the + * program emits only through untyped values or computed names. */ ScrEmitter *scr_emitter_on_flex(ScrEmitter *em, ScrStr *name, const ScrDyn *cb, bool once, bool prepend); +/* Registration and emit through an untyped value (the class dispatch of + * compiled emitters): on_any adds runtime typed adapters under 'error' and + * the meta events and installs the dynamic half; emit_any keeps Node's + * unhandled-'error' contract and gives typed meta listeners a string. */ +ScrEmitter *scr_emitter_on_any(ScrEmitter *em, ScrStr *name, const ScrDyn *cb, + bool once, bool prepend); +bool scr_emitter_emit_any(ScrEmitter *em, ScrStr *name, ScrDyn *const *args, size_t argc); ScrEmitter *scr_emitter_off_dyn(ScrEmitter *em, ScrStr *name, const ScrDyn *cb); +/* The registry's checked-dynamic half (meta-event argument vectors and + * rebuilt listener values), installed by the first on_any call; NULL in + * programs that never register a listener through an untyped value. + * Internal. */ +typedef struct ScrEeDynOps ScrEeDynOps; +extern SCR_TL const ScrEeDynOps *scr_ee_dyn; /* The adapter-mediated registration (the LLVM backend's listeners): the * ADAPTER closure is what emit invokes, the ORIGINAL closure is the * entry's identity (the on_dyn split, with a plain-closure identity). @@ -139,6 +154,12 @@ ScrEmitter *scr_emitter_remove_all(ScrEmitter *em, ScrStr *name, bool all); * emit('error', err): no listener ⇒ THROWS err (borrowed; +1 taken). */ bool scr_emitter_emit(ScrEmitter *em, ScrStr *name, ...); bool scr_emitter_emit_flex(ScrEmitter *em, ScrStr *name, ScrDyn *const *args, size_t argc); +/* An untyped emit of an event with a fixed tuple: the arguments as given + * for dynamic listeners, plus the tuple converted by the caller as + * scr_emitter_emit's variadics for typed listeners. has_typed answers + * whether any listener of the name needs the converted tuple. */ +bool scr_emitter_emit_mixed(ScrEmitter *em, ScrStr *name, ScrDyn *const *args, size_t argc, ...); +bool scr_emitter_has_typed(ScrEmitter *em, ScrStr *name); bool scr_emitter_emit_error_flex(ScrEmitter *em, ScrDyn *error); bool scr_emitter_emit_error(ScrEmitter *em, ScrStr *name, ScrError *err); double scr_emitter_listener_count(ScrEmitter *em, ScrStr *name); @@ -154,6 +175,7 @@ ScrStr *scr_emitter_error_monitor(void); * other read). The stream constructors call this. */ void scr_emitter_reserve(ScrEmitter *em, const char *name); ScrArr *scr_emitter_listeners(ScrEmitter *em, ScrStr *name); /* +1 closures */ +ScrDyn *scr_emitter_listeners_dyn(ScrEmitter *em, ScrStr *name); /* +1 dyn array */ ScrEmitter *scr_emitter_set_max(ScrEmitter *em, double n); /* returns em +1 */ double scr_emitter_get_max(ScrEmitter *em); void scr_emitter_set_default_max(double n); diff --git a/tests/corpus/6580-emitter-typed-untyped-listeners.ts b/tests/corpus/6580-emitter-typed-untyped-listeners.ts new file mode 100644 index 000000000..42f3db549 --- /dev/null +++ b/tests/corpus/6580-emitter-typed-untyped-listeners.ts @@ -0,0 +1,50 @@ +// One native EventEmitter used both through its static type and through an +// untyped reference: typed and untyped listeners of the same event share +// one listener list, so emits from either side reach every listener in +// registration order, with once, prepend, off and listenerCount over the +// whole list. Untyped emits pass their arguments as given (extra ones +// included) to untyped listeners and the event's typed tuple to typed ones. +import { EventEmitter } from "node:events"; + +const e = new EventEmitter(); +const d: any = e; +const log: string[] = []; +const typed = (n: number, s: string): void => { + log.push(`typed ${n} ${s}`); +}; +e.on("tick", typed); +d.on("tick", (...args: any[]) => { + log.push(`untyped ${JSON.stringify(args)}`); +}); +e.once("tick", (n: number) => { + log.push(`typed once ${n}`); +}); +d.once("tick", (n: any) => { + log.push(`untyped once ${n}`); +}); +d.prependListener("tick", () => { + log.push("untyped first"); +}); +console.log(e.listenerCount("tick"), d.listenerCount("tick")); +console.log(e.emit("tick", 1, "a")); +console.log(d.emit("tick", 2, "b", "extra")); +console.log(e.listenerCount("tick"), d.listenerCount("tick"), d.listenerCount("tick", typed)); +d.off("tick", typed); +e.emit("tick", 3, "c"); +d.emit("tick", 4, "d"); +console.log(log.join("\n")); +log.length = 0; + +// A typed prefix listener next to untyped listeners of a longer tuple. +e.on("pair", (a: string) => log.push(`prefix ${a}`)); +d.on("pair", (a: any, b: any) => log.push(`untyped ${a} ${b}`)); +e.emit("pair", "x", 1); +d.emit("pair", "y", 2); +console.log(log.join("\n")); + +// Names only untyped code uses stay untyped, and an emit nobody hears +// answers false. +d.on("loose", (...args: any[]) => console.log("loose", args.length, args)); +console.log(d.emit("loose"), d.emit("loose", null, [1], { k: true })); +console.log(d.emit("nobody", 1), e.emit("tick", 5, "e")); +console.log(d.eventNames(), e.eventNames()); diff --git a/tests/corpus/6581-emitter-untyped-error-and-meta.ts b/tests/corpus/6581-emitter-untyped-error-and-meta.ts new file mode 100644 index 000000000..1cdd10bec --- /dev/null +++ b/tests/corpus/6581-emitter-untyped-error-and-meta.ts @@ -0,0 +1,59 @@ +// 'error', errorMonitor and the newListener/removeListener meta events on +// an emitter used both through its static type and through an untyped +// reference. Untyped emits of 'error' keep Node's contract: listeners of +// either kind receive the payload, and with none an Error throws itself +// while other payloads throw ERR_UNHANDLED_ERROR. Untyped meta listeners +// receive the event name and the listener function. +import { EventEmitter, errorMonitor } from "node:events"; + +const plain: any = new EventEmitter(); +plain.on("newListener", (name: any, fn: any) => console.log("plain newListener", name, typeof fn)); +plain.on("removeListener", (name: any, fn: any) => console.log("plain removeListener", name, fn.name)); +function named(): void {} +plain.on("x", named); +plain.off("x", named); +plain.emit("newListener", "manual", named); + +const e = new EventEmitter(); +const d: any = e; +d.on("newListener", (name: any) => { + console.log("newListener", name); +}); +d.on("removeListener", (name: any) => { + console.log("removeListener", name); +}); +e.on("removeListener", (name: string) => console.log("typed removeListener", name)); +const tap = (n: any): void => console.log("untyped v", n); +d.on("v", tap); +e.on("v", (n: number) => console.log("typed v", n)); +e.emit("v", 1); +d.emit("v", 2); +d.off("v", tap); +e.emit("v", 3); +d.removeAllListeners("v"); +console.log(d.listenerCount("v")); + +d.removeAllListeners("newListener"); +d.removeAllListeners("removeListener"); +const monitor = (err: Error): void => console.log("monitor", err.message); +e.on(errorMonitor, monitor); +e.on("error", (err: Error) => console.log("typed error", err.message)); +d.on("error", (err: any) => console.log("untyped error", err instanceof Error, err.message)); +d.emit("error", new Error("boom")); +e.emit("error", new Error("bang")); +d.removeAllListeners("error"); +try { + d.emit("error", new Error("unhandled")); +} catch (err: any) { + console.log("threw", err instanceof Error, err.message); +} +e.off(errorMonitor, monitor); +for (const payload of ["just text", 42, undefined, null, false]) { + try { + d.emit("error", payload); + } catch (err: any) { + console.log("threw", err.code, err.message); + } +} +d.on("error", (x: any, more: any) => console.log("untyped got", x, more)); +console.log(d.emit("error", 42, "more"), d.listenerCount("error")); diff --git a/tests/corpus/6582-emitter-class-expression-escapes.cjs b/tests/corpus/6582-emitter-class-expression-escapes.cjs new file mode 100644 index 000000000..da641f1b6 --- /dev/null +++ b/tests/corpus/6582-emitter-class-expression-escapes.cjs @@ -0,0 +1,61 @@ +// A function-local class expression extending EventEmitter whose +// instances escape untyped: the class body emits with fixed tuples while +// callers listen, emit and inspect through untyped references. +"use strict"; +const { EventEmitter } = require("node:events"); + +function makeQueue(label) { + const Queue = class extends EventEmitter { + constructor() { + super(); + this.items = []; + } + push(item) { + this.items.push(item); + this.emit("added", item, this.items.length); + return this; + } + drain() { + const items = this.items.splice(0); + this.emit("drained", items.length); + return items; + } + fail(message) { + this.emit("error", new Error(`${label}: ${message}`)); + } + }; + return new Queue(); +} + +const q = makeQueue("q1"); +q.on("added", (item, size) => console.log("added", item, size)); +q.once("added", (item) => console.log("first add", item)); +q.prependListener("drained", (count) => console.log("drained", count)); +q.push("a").push("b"); +q.emit("added", "manual", 99, "extra"); +console.log(q.drain()); +q.on("error", (err) => console.log("error", err.message)); +q.fail("broken"); +console.log(q.listenerCount("added"), q.eventNames(), q.getMaxListeners()); +q.removeAllListeners("added"); +q.push("c"); +console.log(q.listenerCount("added"), q.listeners("drained").length); +q.removeAllListeners(); +console.log(q.eventNames()); + +const q2 = makeQueue("q2"); +try { + q2.fail("nobody listens"); +} catch (err) { + console.log("caught", err.message); +} +const counts = []; +const q3 = makeQueue("q3"); +const count = () => counts.push(q3.listenerCount("added")); +q3.on("added", count); +q3.on("added", function self() { + q3.off("added", self); + counts.push("self"); +}); +q3.push(1).push(2); +console.log(counts); diff --git a/tests/corpus/6583-emitter-untyped-surface.ts b/tests/corpus/6583-emitter-untyped-surface.ts new file mode 100644 index 000000000..18cd3cb91 --- /dev/null +++ b/tests/corpus/6583-emitter-untyped-surface.ts @@ -0,0 +1,47 @@ +// The EventEmitter surface beyond on/once/emit/off called through untyped +// references to compiled emitters (a subclass instance and a plain +// EventEmitter): listenerCount, eventNames, listeners, rawListeners, +// removeAllListeners, setMaxListeners and getMaxListeners answer Node's +// results, and the chaining forms return the emitter. +import { EventEmitter } from "node:events"; + +class Bus extends EventEmitter { + ping(n: number): boolean { + return this.emit("ping", n); + } +} +const b = new Bus(); +const u: unknown = b; +const d = u as any; +const f = (n: any): void => console.log("f", n); +d.on("ping", f); +d.prependOnceListener("ping", (n: any) => console.log("once first", n)); +d.prependListener("ping", (n: any) => console.log("first", n)); +console.log(d.listenerCount("ping"), d.listenerCount("ping", f), d.listenerCount("none")); +console.log(d.eventNames(), d.listenerCount()); +const all = d.listeners("ping"); +console.log(all.length, d.rawListeners("ping").length, all[2] === f, typeof all[0]); +console.log(d.listeners("none")); +console.log(d.getMaxListeners(), d.setMaxListeners(2) === d, b.getMaxListeners()); +console.log(b.ping(1), b.ping(2)); +try { + d.setMaxListeners(-1); +} catch (err: any) { + console.log(err.name, err.code, err.message); +} +try { + d.setMaxListeners("x"); +} catch (err: any) { + console.log(err.name, err.code, err.message); +} +d.on("other", () => {}); +console.log(d.removeAllListeners("ping") === d, d.eventNames()); +console.log(d.removeAllListeners() === d, d.eventNames(), d.listenerCount("other")); +console.log(b.ping(3)); + +const plain: any = new EventEmitter(); +plain.on("x", () => {}); +plain.on("y", () => {}); +console.log(plain.eventNames(), plain.getMaxListeners(), plain.listeners("x").length); +plain.setMaxListeners(0); +console.log(plain.getMaxListeners(), plain.removeAllListeners("x").eventNames()); diff --git a/tests/corpus/6584-emitter-class-body-untyped-listeners.ts b/tests/corpus/6584-emitter-class-body-untyped-listeners.ts new file mode 100644 index 000000000..f2ca78d84 --- /dev/null +++ b/tests/corpus/6584-emitter-class-body-untyped-listeners.ts @@ -0,0 +1,54 @@ +// A class whose methods emit with fixed tuples, observed by listeners +// registered through untyped references: class-instance payloads keep +// their identity, once/off apply across typed and untyped listeners, a +// listener may capture and remove itself, untyped emits reach events.once, +// and errorMonitor sees untyped 'error' emits. +import { EventEmitter, once, errorMonitor } from "node:events"; + +class Job { + readonly tags: string[] = []; + readonly id: number; + constructor(id: number) { + this.id = id; + } +} + +class Worker extends EventEmitter { + run(job: Job): void { + this.emit("job", job, job.tags.length); + } + finish(): void { + this.emit("done"); + } +} + +const w = new Worker(); +const u: any = w; +const first = new Job(7); +u.on("job", (j: any, count: any) => { + console.log("untyped job", j.id, count, j === first, j instanceof Job); + j.tags.push("seen"); +}); +w.on("job", (j: Job) => console.log("typed job", j.id, j.tags.join(","))); +u.once("done", () => console.log("done once")); +w.run(first); +w.finish(); +w.finish(); +u.emit("job", new Job(8), 0); +const self = (): void => { + console.log("self", u.listenerCount("job")); + u.off("job", self); +}; +u.on("job", self); +w.run(new Job(9)); +w.run(new Job(10)); +w.on(errorMonitor, (err: Error) => console.log("monitor", err.message)); +w.on("error", (err: Error) => console.log("typed error", err.message)); +u.emit("error", new Error("e1")); +async function main(): Promise { + const pending = once(w, "job"); + u.emit("job", new Job(11), 1); + const [j, count]: any[] = await pending; + console.log("once resolved", j.id, count); +} +main(); From 0cf62ebecfe8e25cc00971ec2c17988186a5624f Mon Sep 17 00:00:00 2001 From: Malte Ubl Date: Tue, 20 Oct 2026 17:36:37 -0700 Subject: [PATCH 02/51] Answer EventEmitter methods read as values off compiled emitters Reading an EventEmitter.prototype method off a compiled emitter (a program class extending EventEmitter, or new EventEmitter()) through an untyped value answered undefined (`typeof o.on`, `const f = o.on; f.call(o, ...)`, `o.emit.bind(o)`), and EventEmitter.prototype.on in JavaScript was a fence. Such reads, typed value reads, computed keys, `in`, and EventEmitter.prototype. now answer the runtime's EventEmitter.prototype functions, the ones constructor-function emitters inherit, so `o.on === o.addListener === EventEmitter.prototype.on` and names and lengths match Node. The first read installs a program dispatch that those functions call for a compiled emitter receiver; it invokes the method through the existing untyped dispatch with the call's argument count (emit forwards up to 16 arguments after the name, then fences). An untyped emit on, or read of emit from, a class that overrides emit now fences instead of bypassing the override for the native emit. A subclass visited under the boxed EventEmitter capsule keeps its own dispatch plan for values boxed as itself. Co-Authored-By: Claude Opus 5.5 --- .../limitations/builtin-apis/eventemitter.mdx | 5 +- .../compiler/src/backend/llvm/lib-shared.ts | 1 + .../lowering/class-dynamic-dispatch.ts | 67 +++- .../lowering/emitter-method-values.ts | 303 ++++++++++++++++++ .../lowering/expressions/property-access.ts | 10 + .../compiler/src/ir/builtin-signatures.ts | 4 + packages/compiler/src/ir/runtime-features.ts | 2 + ...6600-emitter-method-values-untyped.ts.json | 6 + .../6601-emitter-method-values.cjs.json | 6 + .../6602-emitter-method-values-typed.ts.json | 6 + packages/runtime/src/scr_dyn.h | 5 + packages/runtime/src/scr_dyn_builtin_class.c | 45 ++- .../6600-emitter-method-values-untyped.ts | 75 +++++ tests/corpus/6601-emitter-method-values.cjs | 73 +++++ .../6602-emitter-method-values-typed.ts | 34 ++ 15 files changed, 638 insertions(+), 4 deletions(-) create mode 100644 packages/compiler/src/frontend/lowering/emitter-method-values.ts create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6600-emitter-method-values-untyped.ts.json create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6601-emitter-method-values.cjs.json create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6602-emitter-method-values-typed.ts.json create mode 100644 tests/corpus/6600-emitter-method-values-untyped.ts create mode 100644 tests/corpus/6601-emitter-method-values.cjs create mode 100644 tests/corpus/6602-emitter-method-values-typed.ts diff --git a/docs/content/limitations/builtin-apis/eventemitter.mdx b/docs/content/limitations/builtin-apis/eventemitter.mdx index 0e085c56f..e6c140cde 100644 --- a/docs/content/limitations/builtin-apis/eventemitter.mdx +++ b/docs/content/limitations/builtin-apis/eventemitter.mdx @@ -5,6 +5,9 @@ An untyped `emit` of an event that has a listener with typed parameters converts its arguments to the event's typed tuple before any listener runs, and throws a `TypeError` when they do not convert. Through an untyped value, `listeners()` and `rawListeners()` of an event with a listener that has typed parameters are unsupported, and `emit('error')` of an object that is not an Error with no `error` listener is unsupported. Untyped `newListener` and `removeListener` listeners receive `undefined` in place of a listener registered with typed parameters. - Reading EventEmitter methods as properties through an untyped value (`typeof emitter.listeners`) answers `undefined`. + An untyped `emit` on, or an untyped read of `emit` from, an instance of a class that overrides `emit` is unsupported. + Calling `EventEmitter.prototype.emit` as a function value on a compiled emitter with more than 16 arguments after the event name is unsupported. + Calling EventEmitter.prototype's methods with an instance of a compiled class that does not extend EventEmitter as the receiver throws a `TypeError`. + Through an untyped value, a stream's `emit` and listener-introspection methods read as `undefined`. `events.once` accepts EventEmitter and stream instances with a literal event name and an optional object-literal `{ signal }`; a stream's `data` event, other options values, and EventTarget emitters are unsupported. Reading `defaultMaxListeners` through a CommonJS destructured or member binding (`const { defaultMaxListeners } = require("events")`) is unsupported. diff --git a/packages/compiler/src/backend/llvm/lib-shared.ts b/packages/compiler/src/backend/llvm/lib-shared.ts index 007c78c9e..3f071d27c 100644 --- a/packages/compiler/src/backend/llvm/lib-shared.ts +++ b/packages/compiler/src/backend/llvm/lib-shared.ts @@ -1465,6 +1465,7 @@ export const LIB_FN_SYMS: Record = { "dyn.classInherit": "scr_dyn_class_inherit", "dyn.builtinClass": "scr_dyn_builtin_class", "dyn.builtinStreamClass": "scr_dyn_builtin_stream_class", + "dyn.emitterMethod": "scr_dyn_emitter_method", "dyn.builtinInstanceOf": "scr_dyn_builtin_instanceof", "dyn.functionInstanceOf": "scr_dyn_function_instanceof", "dyn.classSuper": "scr_dyn_class_super", diff --git a/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts b/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts index c9e6768a6..af2a61213 100644 --- a/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts +++ b/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts @@ -74,6 +74,12 @@ import { lowerDynamicEmitterOn, } from "./lower-event-emitter.js"; import { protocolArm, protocolFieldArm } from "./class-protocol-dispatch.js"; +import { + EMITTER_PROTOTYPE_METHODS, + emitterMethodValue, + inheritedEmitterMethod, + overriddenEmit, +} from "./emitter-method-values.js"; import { STREAM_OPERATORS } from "./lower-stream.js"; import { deletedPropertyNames, fixedFieldFamily } from "./js-class-fields.js"; @@ -2771,6 +2777,7 @@ export class ClassDynamicDispatch { !isClassOwnEnumerableFieldName(dispatch.name) || (!(info.builtinStream !== undefined && streamProperty(dispatch.name)) && !eventTargetInheritedMember(info, dispatch.name) && + !inheritedEmitterMethod(lowerer, info, dispatch.name) && !findMethodOn(lowerer, info, dispatch.name) && !findGenericMethodOn(lowerer, info, dispatch.name) && dispatch.name !== "constructor" && @@ -2889,6 +2896,12 @@ export class ClassDynamicDispatch { loc, }, ]; + // An inherited EventEmitter.prototype method: `in` sees it, the + // own-property probes see only an own property that shadows it. + if (dispatch.probe === "in" && inheritedEmitterMethod(lowerer, info, name)) + return [ + { kind: "return", value: { kind: "boolLit", value: true, type: BOOL, loc }, loc }, + ]; if (!dispatch.probe) return dispatch.reflect ? this.reflectPropertyBody(lowerer, { ...dispatch, name }, info, receiver) @@ -3101,6 +3114,8 @@ export class ClassDynamicDispatch { for (const name of [...STREAM_BOOL_PROPERTIES, ...STREAM_NUM_PROPERTIES]) names.add(name); const inherited = eventTargetApiMembers(info); if (inherited) for (const name of inherited) names.add(name); + for (const name of EMITTER_PROTOTYPE_METHODS.keys()) + if (inheritedEmitterMethod(lowerer, info, name)) names.add(name); for (let owner: ClassInfo | null = info; owner; owner = owner.base) { const methodNames = [...owner.methods.keys()]; @@ -3548,6 +3563,10 @@ export class ClassDynamicDispatch { for (const name of c.methods.keys()) names.add(name.startsWith("get:") || name.startsWith("set:") ? name.slice(4) : name); } + // EventEmitter.prototype's methods on compiled emitters. + for (const { info } of this.propertyReceivers(lowerer, true)) + for (const name of EMITTER_PROTOTYPE_METHODS.keys()) + if (inheritedEmitterMethod(lowerer, info, name)) names.add(name); return names; } @@ -4308,7 +4327,9 @@ export class ClassDynamicDispatch { (!canonical || !seen.has(info.def.name)) && (info === capsule || lowerer.classCanBeConstructed(info)) ) { - seen.add(info.def.name); + // A subclass visited under a runtime capsule (boxed as that + // base) keeps its own canonical plan for values boxed as itself. + if (canonical) seen.add(info.def.name); plans.push({ info, capsule: canonical ? info : capsule, @@ -4728,6 +4749,45 @@ export class ClassDynamicDispatch { loc, }, ]; + // An emit override has no function value (it is specialized per event + // name and payload type). + if ( + !write && + name === "emit" && + overriddenEmit(info) && + inheritedEmitterMethod(lowerer, info, name) + ) + return [ + { + kind: "runtimeFence", + code: "SC2020", + message: + "reading an overridden EventEmitter emit through an untyped value is not supported yet", + loc, + }, + ]; + // An inherited EventEmitter.prototype method: an own property shadows + // it, otherwise the read answers the function + // `EventEmitter.prototype.` reads. + if (!write && inheritedEmitterMethod(lowerer, info, name)) { + const instance = lowerer.coerceToExpected(receiver, DYN); + const bag = this.readPropertyBagOrEmpty(lowerer, instance, loc); + const key: IrExpr = { kind: "strLit", value: name, type: STRING, loc }; + return [ + { + kind: "return", + value: { + kind: "ternary", + cond: this.readBagHas(lowerer, "dyn.hasKey", bag, key, loc), + then: { kind: "libCall", fn: "dyn.bagGet", args: [bag, key, instance], type: DYN, loc }, + else_: emitterMethodValue(lowerer, name, loc), + type: DYN, + loc, + }, + loc, + }, + ]; + } const field = info.fields.get(name); const member = `${write ? "set" : "get"}:${name}`; const accessor = field ? null : findMethodOn(lowerer, info, member); @@ -5159,6 +5219,11 @@ export class ClassDynamicDispatch { loc, }, ]; + // An emit override is specialized per event name and payload type; an + // untyped emit names neither, so it refuses rather than bypassing the + // override for the native emit. + if (method === "emit" && overriddenEmit(info) && inheritedEmitterMethod(lowerer, info, method)) + return fence(); if ( emitterRooted(lowerer, info) && !streamRooted(info) && diff --git a/packages/compiler/src/frontend/lowering/emitter-method-values.ts b/packages/compiler/src/frontend/lowering/emitter-method-values.ts new file mode 100644 index 000000000..f04957e11 --- /dev/null +++ b/packages/compiler/src/frontend/lowering/emitter-method-values.ts @@ -0,0 +1,303 @@ +/** + * EventEmitter.prototype's methods as values. A read of `o.on` on a + * compiled emitter (a program class extending EventEmitter, or + * `new EventEmitter()`) through an untyped value or as a value, and + * `EventEmitter.prototype.on` itself, answer the runtime's + * EventEmitter.prototype functions (scr_dyn_builtin_class.c), the same + * functions JavaScript constructor functions inherit: `on` and + * `addListener` are one function, as are `off` and `removeListener`, and + * each carries Node's `name` and `length`. Called on a compiled emitter + * (`f.call(o, …)`, `f.bind(o)`, `f.apply(o, …)`), such a function hands + * the call to the program's dispatch below, which invokes the method on + * the receiver through the same untyped dispatch a direct `o.on(…)` call + * reaches. + */ +import { BOOL, DYN, F64, STRING, type IrExpr, type IrStmt, type SrcLoc } from "../../ir/ir.js"; +import { dynUndefinedExpr, strLit, varRef } from "../../ir/build.js"; +import * as ts from "../ts7/adapter.js"; +import { locOf } from "../program.js"; +import type { Lowerer } from "./lowerer.js"; +import { findGenericMethodOn, findMethodOn, type ClassInfo } from "./lower-classes.js"; + +/** EventEmitter.prototype's method names. */ +export const EMITTER_PROTOTYPE_METHODS: ReadonlySet = new Set([ + "setMaxListeners", + "getMaxListeners", + "emit", + "addListener", + "on", + "prependListener", + "once", + "prependOnceListener", + "removeListener", + "off", + "removeAllListeners", + "listeners", + "rawListeners", + "listenerCount", + "eventNames", +]); + +/** `emit` forwards this many arguments after the event name; a call with + * more fences. */ +const EMIT_ARGUMENTS = 16; + +/** The runtime's SCR_EE_M_* order: each method's name and the argument + * counts its call forwards. Node's methods that look at + * `arguments.length` (removeAllListeners, emit, listenerCount's optional + * listener) get exactly the arguments given; the rest get their declared + * parameters, an absent one as undefined. */ +const DISPATCHED: readonly { + method: string; + fixed?: number; + counts?: (given: number) => number; +}[] = [ + { method: "addListener", fixed: 2 }, + { method: "prependListener", fixed: 2 }, + { method: "once", fixed: 2 }, + { method: "prependOnceListener", fixed: 2 }, + { method: "removeListener", fixed: 2 }, + { method: "removeAllListeners", counts: (given) => Math.min(given, 1) }, + { method: "emit", counts: (given) => Math.max(given, 1) }, + { method: "listeners", fixed: 1 }, + { method: "rawListeners", fixed: 1 }, + { method: "listenerCount", counts: (given) => (given < 2 ? 1 : 2) }, + { method: "eventNames", fixed: 0 }, + { method: "setMaxListeners", fixed: 1 }, + { method: "getMaxListeners", fixed: 0 }, +]; + +const DISPATCH = "%emitter.method.dispatch"; + +/** An EventEmitter.prototype method a non-stream emitter class inherits + * (no field, method or accessor of that name in its chain; streams keep + * their own surface). An `emit` override is not a method, so `emit` counts + * here too; overriddenEmit separates it. */ +export function inheritedEmitterMethod(lowerer: Lowerer, info: ClassInfo, name: string): boolean { + if (!EMITTER_PROTOTYPE_METHODS.has(name)) return false; + let emitter = false; + for (let c: ClassInfo | null = info; c; c = c.base) { + if (c.builtinStream !== undefined) return false; + if (c.builtinEmitter) emitter = true; + } + return ( + emitter && + !info.fields.has(name) && + !findMethodOn(lowerer, info, name) && + !findGenericMethodOn(lowerer, info, name) && + !findMethodOn(lowerer, info, `get:${name}`) && + !findMethodOn(lowerer, info, `set:${name}`) + ); +} + +/** True when the class or an ancestor overrides `emit` (lowered as + * per-event specializations rather than a method). */ +export function overriddenEmit(info: ClassInfo): boolean { + for (let c: ClassInfo | null = info; c; c = c.base) if (c.emitOverride) return true; + return false; +} + +/** A typed read of an inherited EventEmitter.prototype method as a value + * (`const f = emitter.on`, `typeof emitter.emit`) on a compiled emitter or + * a union of them: the receiver is evaluated, the read answers the + * interned method value. */ +export function lowerEmitterInstanceMethod( + lowerer: Lowerer, + expr: ts.PropertyAccessExpression, +): IrExpr | null { + const name = expr.name.text; + if (!EMITTER_PROTOTYPE_METHODS.has(name) || expr.questionDotToken) return null; + if (ts.isCallExpression(expr.parent) && expr.parent.expression === expr) return null; + const symbol = lowerer.checker.getSymbolAtLocation(expr.name); + if (!symbol || !lowerer.isStdlibSymbol(symbol)) return null; + const receiverType = lowerer.mapTypeOf(lowerer.typeOf(expr.expression)); + if (!receiverType) return null; + const arms = + receiverType.kind === "union" + ? (lowerer.unions.get(receiverType.unionId)?.arms ?? []) + : [receiverType]; + if ( + !arms.length || + !arms.every((arm) => { + const info = arm.kind === "object" ? lowerer.classes.get(arm.className) : undefined; + return info !== undefined && inheritedEmitterMethod(lowerer, info, name); + }) + ) + return null; + const loc = locOf(expr); + return { + kind: "seqExpr", + stmts: [{ kind: "exprStmt", expr: lowerer.lowerExpr(expr.expression), loc }], + result: emitterMethodValue(lowerer, name, loc), + type: DYN, + loc, + }; +} + +/** The program's dispatch for EventEmitter.prototype's functions on + * compiled emitters: `this` is the emitter, the first argument the + * method's index, the rest the call's arguments. */ +function emitterMethodDispatch(lowerer: Lowerer, loc: SrcLoc): IrExpr { + if (!lowerer.liftedFns.some((fn) => fn.name === DISPATCH)) { + const self = varRef("self", DYN, loc); + const values = varRef("values", DYN, loc); + const index = varRef("index", F64, loc); + const given = varRef("given", F64, loc); + const number = (value: number): IrExpr => ({ kind: "numLit", value, type: F64, loc }); + const equals = (left: IrExpr, value: number): IrExpr => ({ + kind: "bin", + op: "===", + left, + right: number(value), + type: BOOL, + loc, + }); + const element = (i: number): IrExpr => ({ + kind: "dynKeyGet", + value: values, + key: lowerer.coerceToExpected(number(i), DYN), + type: DYN, + loc, + }); + const invoke = (method: string, n: number): IrStmt => ({ + kind: "return", + value: { + kind: "dynInvoke", + recv: self, + method, + calleeName: method, + args: Array.from({ length: n }, (_, i) => element(i + 1)), + type: DYN, + loc, + }, + loc, + }); + const body: IrStmt[] = [ + { + kind: "varDecl", + localId: "self", + init: { kind: "libCall", fn: "dyn.this", args: [], type: DYN, loc }, + loc, + }, + { + kind: "varDecl", + localId: "index", + init: { kind: "dynCheck", value: element(0), type: F64, loc }, + loc, + }, + { + kind: "varDecl", + localId: "given", + init: { + kind: "bin", + op: "-", + left: { + kind: "dynCheck", + value: { kind: "dynKeyGet", value: values, key: strLit("length", loc), type: DYN, loc }, + type: F64, + loc, + }, + right: number(1), + type: F64, + loc, + }, + loc, + }, + ]; + DISPATCHED.forEach(({ method, fixed, counts }, i) => { + const arms: IrStmt[] = []; + if (fixed !== undefined || !counts) arms.push(invoke(method, fixed ?? 0)); + else { + // One arm per argument count up to the last one that matters. + const limit = method === "emit" ? EMIT_ARGUMENTS + 1 : 2; + for (let n = 0; n < limit; n++) + arms.push({ + kind: "if", + cond: equals(given, n), + then: [invoke(method, counts(n))], + else_: null, + loc, + }); + if (method === "emit") + arms.push( + { + kind: "if", + cond: equals(given, limit), + then: [invoke(method, limit)], + else_: null, + loc, + }, + { + kind: "runtimeFence", + code: "SC2020", + message: `calling EventEmitter.prototype.emit through a method value with more than ${EMIT_ARGUMENTS} arguments after the event name is not supported yet`, + loc, + }, + ); + else arms.push(invoke(method, counts(limit))); + } + body.push({ kind: "if", cond: equals(index, i), then: arms, else_: null, loc }); + }); + body.push({ kind: "return", value: dynUndefinedExpr(loc), loc }); + lowerer.liftedFns.push({ + name: DISPATCH, + params: [{ localId: "values", name: "values", type: DYN }], + returnType: DYN, + locals: [ + { id: "values", name: "values", type: DYN, mutable: false }, + { id: "self", name: "self", type: DYN, mutable: false }, + { id: "index", name: "index", type: F64, mutable: false }, + { id: "given", name: "given", type: F64, mutable: false }, + ], + body, + loc, + }); + } + return { + kind: "dynFrom", + value: { + kind: "closure", + fnName: DISPATCH, + captures: [], + type: { kind: "func", params: [], ret: DYN, rest: true, argumentsAll: true }, + loc, + }, + type: DYN, + loc, + }; +} + +/** EventEmitter.prototype[name] (DYN). */ +export function emitterMethodValue(lowerer: Lowerer, name: string, loc: SrcLoc): IrExpr { + return { + kind: "libCall", + fn: "dyn.emitterMethod", + args: [{ kind: "strLit", value: name, type: STRING, loc }, emitterMethodDispatch(lowerer, loc)], + type: DYN, + loc, + }; +} + +/** `EventEmitter.prototype.` (any import spelling of the class): + * the interned method value. */ +export function lowerEmitterPrototypeMethod( + lowerer: Lowerer, + expr: ts.PropertyAccessExpression, +): IrExpr | null { + if (!EMITTER_PROTOTYPE_METHODS.has(expr.name.text)) return null; + const prototype = expr.expression; + if ( + !ts.isPropertyAccessExpression(prototype) || + prototype.questionDotToken || + prototype.name.text !== "prototype" + ) + return null; + const owner = prototype.expression; + const symbol = ts.isIdentifier(owner) + ? lowerer.resolveValueSymbol(owner) + : ts.isPropertyAccessExpression(owner) + ? lowerer.checker.getSymbolAtLocation(owner.name) + : null; + if (!lowerer.builtinEmitterInfoOf(symbol)) return null; + return emitterMethodValue(lowerer, expr.name.text, locOf(expr)); +} diff --git a/packages/compiler/src/frontend/lowering/expressions/property-access.ts b/packages/compiler/src/frontend/lowering/expressions/property-access.ts index 36ea0e8c2..1e2722114 100644 --- a/packages/compiler/src/frontend/lowering/expressions/property-access.ts +++ b/packages/compiler/src/frontend/lowering/expressions/property-access.ts @@ -128,6 +128,10 @@ import { lowerReadlineProperty } from "../builtins/readline.js"; import { lowerEventStaticProperty } from "../builtins/numbers.js"; import { lowerStreamProperty, lowerStreamStateProperty, streamSidesOf } from "../lower-stream.js"; import { lowerEventObjectProperty } from "../lower-event-target.js"; +import { + lowerEmitterInstanceMethod, + lowerEmitterPrototypeMethod, +} from "../emitter-method-values.js"; import { iteratorResultValueRead } from "../lower-generators.js"; import { lowerGlobalValue } from "../lower-global-value.js"; import { nullishGenericBindingUnitOf } from "../binding-analysis.js"; @@ -329,6 +333,12 @@ export function lowerPropertyAccessExpr( const ctor = runtimeConstructorReceiver(lowerer, expr); if (ctor) return lowerRuntimeConstructor(lowerer, expr, ctor.info); } + // EventEmitter.prototype's methods as values (emitter-method-values.ts). + { + const method = + lowerEmitterPrototypeMethod(lowerer, expr) ?? lowerEmitterInstanceMethod(lowerer, expr); + if (method) return method; + } // `C.name` through a runtime-provided class VALUE (a `typeof Readable` // parameter): the class object's stored name. if ( diff --git a/packages/compiler/src/ir/builtin-signatures.ts b/packages/compiler/src/ir/builtin-signatures.ts index d181927fc..9bba57167 100644 --- a/packages/compiler/src/ir/builtin-signatures.ts +++ b/packages/compiler/src/ir/builtin-signatures.ts @@ -3611,6 +3611,10 @@ export const LIB_FN_SIGS = defineLibFnSignatures({ * functions over them get native stream state * (scr_dyn_builtin_stream.c). */ "dyn.builtinStreamClass": { argTypes: [F64], result: DYN }, + /** EventEmitter.prototype[name] off EventEmitter (0) above; the first + * read installs the program's dispatch (the DYN function) that runs + * those methods on compiled emitters. */ + "dyn.emitterMethod": { argTypes: [STRING, DYN], result: DYN }, /** `value instanceof C` for those constructors, over the value's * prototype chain. */ "dyn.builtinInstanceOf": { argTypes: [DYN, F64], result: BOOL }, diff --git a/packages/compiler/src/ir/runtime-features.ts b/packages/compiler/src/ir/runtime-features.ts index fd7c98cf6..5c38b14f9 100644 --- a/packages/compiler/src/ir/runtime-features.ts +++ b/packages/compiler/src/ir/runtime-features.ts @@ -259,6 +259,8 @@ const BUILTIN_CLASS_LIB_FNS: readonly IrLibFn[] = [ "dyn.builtinClass", // the stream classes, through scr_dyn_builtin_stream.c "dyn.builtinStreamClass", + // EventEmitter.prototype's methods read off compiled emitters + "dyn.emitterMethod", "dyn.builtinInstanceOf", "dyn.functionInstanceOf", ]; diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6600-emitter-method-values-untyped.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6600-emitter-method-values-untyped.ts.json new file mode 100644 index 000000000..260282e75 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6600-emitter-method-values-untyped.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6600-emitter-method-values-untyped.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6601-emitter-method-values.cjs.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6601-emitter-method-values.cjs.json new file mode 100644 index 000000000..13f1aaa7d --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6601-emitter-method-values.cjs.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6601-emitter-method-values.cjs" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6602-emitter-method-values-typed.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6602-emitter-method-values-typed.ts.json new file mode 100644 index 000000000..409f1eaa9 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6602-emitter-method-values-typed.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6602-emitter-method-values-typed.ts" + ], + "diags": [] +} diff --git a/packages/runtime/src/scr_dyn.h b/packages/runtime/src/scr_dyn.h index 401bf2b39..14734ab31 100644 --- a/packages/runtime/src/scr_dyn.h +++ b/packages/runtime/src/scr_dyn.h @@ -1159,6 +1159,11 @@ ScrDyn *scr_dyn_new_class(ScrClassObj *cls, const char *type_key); ScrDyn *scr_dyn_builtin_class(double kind); bool scr_dyn_builtin_instanceof(ScrDyn *value, double kind); bool scr_dyn_function_instanceof(ScrDyn *value, ScrDyn *callee); +/* EventEmitter.prototype[name] (+1); the first call installs DISPATCH, + * the program's function that runs those methods on compiled emitters + * (typed refs): called with the emitter as `this`, the method's + * SCR_EE_M_* index, then the call's arguments. */ +ScrDyn *scr_dyn_emitter_method(const ScrStr *name, ScrDyn *dispatch); /* The stream half of those constructors (scr_dyn_builtin_stream.c, linked * with the dynBuiltinStream feature), registered with * scr_dyn_builtin_class_streams before the first stream class value is diff --git a/packages/runtime/src/scr_dyn_builtin_class.c b/packages/runtime/src/scr_dyn_builtin_class.c index 2f947048a..70a67c559 100644 --- a/packages/runtime/src/scr_dyn_builtin_class.c +++ b/packages/runtime/src/scr_dyn_builtin_class.c @@ -66,9 +66,35 @@ static ScrDyn *bc_class(int kind); /* The stream half, once registered (scr_dyn_builtin_stream.c). */ static SCR_TL const ScrDynStreamOps *bc_stream_ops; -/* EventEmitter.prototype's method on a receiver with stream state. */ +/* The program's dispatch for compiled emitters (a program class extending + * EventEmitter, or `new EventEmitter()`), which reach these methods as + * typed refs whose listeners live in the native registry; installed by + * scr_dyn_emitter_method. */ +static SCR_TL ScrDyn *bc_compiled_dispatch; + +/* method(...args) on a compiled emitter: the dispatch runs with the + * receiver as `this` and the method's SCR_EE_M_* index before the + * arguments. */ +static ScrDyn *bc_compiled_event(int method, ScrDyn *self, ScrDyn *const *args, size_t argc) { + ScrDyn **argv = malloc((argc + 1) * sizeof *argv); + if (!argv) scr_trap("out of memory"); + argv[0] = scr_dyn_new_num(method); + for (size_t i = 0; i < argc; i++) argv[i + 1] = args[i]; + scr_dyn_this_push_dyn(self); + ScrDyn *result = scr_dyn_call(bc_compiled_dispatch, argv, argc + 1, bc_names[BC_EMITTER]); + scr_dyn_this_pop(); + scr_dyn_release(argv[0]); + free(argv); + return result; +} + +/* EventEmitter.prototype's method on a receiver with stream state, or on + * a compiled emitter. */ static bool bc_stream_event(int method, ScrDyn *self, ScrDyn *const *args, size_t argc, ScrDyn **result) { - return bc_stream_ops != NULL && bc_stream_ops->event(method, self, args, argc, result); + if (bc_stream_ops != NULL && bc_stream_ops->event(method, self, args, argc, result)) return true; + if (bc_compiled_dispatch == NULL || self->kind != SCR_DYN_TYPED_REF) return false; + *result = bc_compiled_event(method, self, args, argc); + return true; } /* ── property helpers ───────────────────────────────────────────────── */ @@ -1436,6 +1462,8 @@ static void bc_teardown(void) { if (bc_symbols[i]) scr_dyn_release(bc_symbols[i]); bc_symbols[i] = NULL; } + if (bc_compiled_dispatch) scr_dyn_release(bc_compiled_dispatch); + bc_compiled_dispatch = NULL; if (scr_exc_pending()) scr_exc_clear(); } @@ -1542,6 +1570,19 @@ ScrDyn *scr_dyn_builtin_class(double kind) { return scr_dyn_retain(bc_class(index)); } +/* EventEmitter.prototype[name], +1, read off a compiled emitter or the + * class; the first read installs the program's dispatch for compiled + * emitters (borrowed, retained). */ +ScrDyn *scr_dyn_emitter_method(const ScrStr *name, ScrDyn *dispatch) { + ScrDyn *ctor = bc_class(BC_EMITTER); + if (!bc_compiled_dispatch) bc_compiled_dispatch = scr_dyn_retain(dispatch); + ScrDyn *prototype = bc_get(ctor, "prototype"); + if (!prototype) return NULL; + ScrDyn *method = bc_get(prototype, name->data); + scr_dyn_release(prototype); + return method; +} + /* OrdinaryHasInstance over the value's prototype chain. */ static bool bc_chain_has(ScrDyn *value, ScrDyn *prototype) { if (value->kind != SCR_DYN_OBJ && value->kind != SCR_DYN_ARR && value->kind != SCR_DYN_FUNC && diff --git a/tests/corpus/6600-emitter-method-values-untyped.ts b/tests/corpus/6600-emitter-method-values-untyped.ts new file mode 100644 index 000000000..1675c97d6 --- /dev/null +++ b/tests/corpus/6600-emitter-method-values-untyped.ts @@ -0,0 +1,75 @@ +// EventEmitter.prototype's methods read as values through untyped +// references to compiled emitters: a program class extending EventEmitter +// and `new EventEmitter()` seen through any and unknown. +import { EventEmitter } from "node:events"; + +class Bus extends EventEmitter { + label = "bus"; + ping(): string { + return `${this.label} ping`; + } +} + +const bus = new Bus(); +const o: any = bus; +console.log(typeof o.on, typeof o.emit, typeof o.listeners, typeof o.ping); +console.log(o.on === o.addListener, o.off === o.removeListener, o.on === o.once); +console.log(o.on === EventEmitter.prototype.on, o.emit === EventEmitter.prototype.emit); +console.log(o.on.name, o.off.name, o.emit.name, o.once.name); +console.log(o.on.length, o.emit.length, o.listeners.length, o.eventNames.length); + +// call, apply and bind with the emitter as the receiver +const add = o.on; +console.log(add.call(o, "a", (x: unknown, y: unknown) => console.log("a:", x, y)) === o); +console.log(o.emit.apply(o, ["a", 1, 2])); +const emit = o.emit.bind(o); +console.log(emit("a", "x"), emit("missing")); +console.log(o.listenerCount.call(o, "a"), o.eventNames.call(o)); +const once = o.once; +once.call(o, "b", () => console.log("b once")); +console.log(emit("b"), emit("b")); +const prepend = o.prependListener; +prepend.call(o, "a", () => console.log("a first")); +emit("a", 3, 4); +console.log(Reflect.apply(o.listenerCount, o, ["a"])); + +// the argument count matters to removeAllListeners and emit +o.on("c", () => {}); +o.on("d", () => {}); +console.log(o.removeAllListeners.call(o, "c") === o, o.eventNames()); +o.removeAllListeners.call(o); +console.log(o.eventNames()); +o.on("many", (...args: unknown[]) => console.log("many:", args.length, args.join(","))); +console.log(emit("many", 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11)); +console.log(o.setMaxListeners.call(o, 3) === o, o.getMaxListeners.call(o)); + +// off through a detached method value +const listener = () => console.log("never"); +o.on("e", listener); +const off = o.off; +off.call(o, "e", listener); +console.log(o.emit("e"), o.listenerCount("e")); + +// membership: inherited, not own +console.log("on" in o, "emit" in o, Object.prototype.hasOwnProperty.call(o, "on")); +const key = ["emit", "listeners"][Number(o.listenerCount("a") > 5)]!; +console.log(typeof o[key], o[key] === EventEmitter.prototype.emit); + +// the class's own members still answer +console.log(o.ping(), o.ping.call(o)); + +// new EventEmitter() through unknown +const plain = new EventEmitter(); +const seen: unknown = plain; +const p = seen as any; +const pOn = p.on; +pOn.call(p, "z", (v: unknown) => console.log("z:", v)); +console.log(p.emit.call(p, "z", 9), p.listeners.call(p, "z").length); +console.log(p.on === o.on, p.rawListeners === EventEmitter.prototype.rawListeners); + +// a detached method called without a receiver throws a TypeError +try { + add("x", () => {}); +} catch (err) { + console.log("detached:", err instanceof TypeError); +} diff --git a/tests/corpus/6601-emitter-method-values.cjs b/tests/corpus/6601-emitter-method-values.cjs new file mode 100644 index 000000000..7873b5f8e --- /dev/null +++ b/tests/corpus/6601-emitter-method-values.cjs @@ -0,0 +1,73 @@ +"use strict"; +// EventEmitter.prototype's methods as values in JavaScript: read off a +// compiled subclass, off `new EventEmitter()`, off EventEmitter.prototype +// by name, and shared with constructor-function emitters. +const EventEmitter = require("events"); +const util = require("util"); + +class Queue extends EventEmitter { + constructor() { + super(); + this.items = []; + } + push(v) { + this.items.push(v); + this.emit("push", v); + return this; + } +} + +const q = new Queue(); +const on = q.on; +on.call(q, "push", (v) => console.log("pushed", v)); +q.push(1).push(2); +for (const m of ["on", "addListener", "emit", "off", "once", "prependOnceListener", "listeners", "listenerCount", "eventNames"]) { + console.log(m, typeof q[m], q[m] === EventEmitter.prototype[m], q[m].name, q[m].length); +} + +// EventEmitter.prototype's functions called on compiled emitters +const proto = EventEmitter.prototype; +proto.on.call(q, "drain", () => console.log("drained")); +console.log(proto.emit.call(q, "drain"), proto.listenerCount.call(q, "push")); +console.log(proto.eventNames.call(q)); +proto.removeAllListeners.call(q, "drain"); +console.log(q.eventNames(), Reflect.apply(proto.listenerCount, q, ["drain"])); + +// new EventEmitter() +const plain = new EventEmitter(); +const emit = plain.emit; +plain.on("hi", (a, b, c) => console.log("hi", a, b, c)); +console.log(emit.call(plain, "hi", 1, "two", [3])); +console.log(emit.call(plain, "hi", 1, 2, 3, 4, 5, 6, 7, 8, 9)); +const fn = () => console.log("never"); +plain.once("x", fn); +const off = plain.off; +off.call(plain, "x", fn); +console.log(plain.emit("x"), plain.listenerCount("x")); +const bound = plain.prependListener.bind(plain); +bound("hi", () => console.log("hi first")); +plain.emit("hi", "a"); + +// a constructor-function emitter shares the same functions +function Legacy() { + EventEmitter.call(this); +} +util.inherits(Legacy, EventEmitter); +const legacy = new Legacy(); +console.log(legacy.on === q.on, legacy.emit === plain.emit); +legacy.on("tick", (n) => console.log("legacy tick", n)); +q.on.call(legacy, "tick", (n) => console.log("legacy tick again", n)); +console.log(plain.emit.call(legacy, "tick", 7), legacy.listenerCount("tick")); + +// method values of a class expression whose instances escape +function makeEmitter() { + return new (class extends EventEmitter { + fire() { + return this.emit("fire", "local"); + } + })(); +} +const local = makeEmitter(); +const localOn = local.on; +localOn.call(local, "fire", (v) => console.log("fire", v)); +console.log(local.fire(), "on" in local, Object.keys(local).includes("on")); diff --git a/tests/corpus/6602-emitter-method-values-typed.ts b/tests/corpus/6602-emitter-method-values-typed.ts new file mode 100644 index 000000000..45fb7f1fc --- /dev/null +++ b/tests/corpus/6602-emitter-method-values-typed.ts @@ -0,0 +1,34 @@ +// EventEmitter.prototype's methods read as values from typed references +// and from EventEmitter.prototype in TypeScript. +import { EventEmitter } from "node:events"; +import * as events from "node:events"; + +class Counter extends EventEmitter { + count = 0; + bump(): void { + this.count++; + this.emit("bump", this.count); + } +} + +const counter = new Counter(); +const plain = new EventEmitter(); +console.log(typeof counter.on, typeof plain.emit, typeof EventEmitter.prototype.listeners); +console.log(counter.on === plain.on, counter.addListener === EventEmitter.prototype.on); +console.log(events.EventEmitter.prototype.off === plain.removeListener); + +const onValue: unknown = counter.on; +const emitValue: unknown = plain.emit; +console.log(typeof onValue, onValue === EventEmitter.prototype.addListener, emitValue === counter.emit); + +const register = counter.on as any; +register.call(counter, "bump", (n: number) => console.log("bump", n)); +counter.bump(); +EventEmitter.prototype.on.call(counter, "bump", (n: number) => console.log("bump again", n)); +counter.bump(); +console.log(EventEmitter.prototype.listenerCount.call(counter, "bump")); +console.log(EventEmitter.prototype.emit.call(plain, "nothing")); + +const pick = (emitter: EventEmitter, name: "on" | "emit"): unknown => + name === "on" ? emitter.on : emitter.emit; +console.log(pick(counter, "on") === pick(plain, "on"), pick(counter, "emit") === pick(plain, "on")); From 3a27a92cb4549ce6cbe58f89efcac9ff8b7b6cb9 Mon Sep 17 00:00:00 2001 From: Malte Ubl Date: Tue, 20 Oct 2026 18:22:43 -0700 Subject: [PATCH 03/51] Forward at most four emit arguments through EventEmitter method values The dispatch instantiates one bridged emit per argument count and emitter class; capping it at four keeps programs that only read emitter members through computed keys from growing by the full table. Co-Authored-By: Claude Opus 5.5 --- docs/content/limitations/builtin-apis/eventemitter.mdx | 2 +- .../compiler/src/frontend/lowering/emitter-method-values.ts | 2 +- tests/corpus/6600-emitter-method-values-untyped.ts | 2 +- tests/corpus/6601-emitter-method-values.cjs | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/content/limitations/builtin-apis/eventemitter.mdx b/docs/content/limitations/builtin-apis/eventemitter.mdx index e6c140cde..8539b814f 100644 --- a/docs/content/limitations/builtin-apis/eventemitter.mdx +++ b/docs/content/limitations/builtin-apis/eventemitter.mdx @@ -6,7 +6,7 @@ Through an untyped value, `listeners()` and `rawListeners()` of an event with a listener that has typed parameters are unsupported, and `emit('error')` of an object that is not an Error with no `error` listener is unsupported. Untyped `newListener` and `removeListener` listeners receive `undefined` in place of a listener registered with typed parameters. An untyped `emit` on, or an untyped read of `emit` from, an instance of a class that overrides `emit` is unsupported. - Calling `EventEmitter.prototype.emit` as a function value on a compiled emitter with more than 16 arguments after the event name is unsupported. + Calling `EventEmitter.prototype.emit` as a function value on a compiled emitter with more than 4 arguments after the event name is unsupported. Calling EventEmitter.prototype's methods with an instance of a compiled class that does not extend EventEmitter as the receiver throws a `TypeError`. Through an untyped value, a stream's `emit` and listener-introspection methods read as `undefined`. `events.once` accepts EventEmitter and stream instances with a literal event name and an optional object-literal `{ signal }`; a stream's `data` event, other options values, and EventTarget emitters are unsupported. diff --git a/packages/compiler/src/frontend/lowering/emitter-method-values.ts b/packages/compiler/src/frontend/lowering/emitter-method-values.ts index f04957e11..7b4992bee 100644 --- a/packages/compiler/src/frontend/lowering/emitter-method-values.ts +++ b/packages/compiler/src/frontend/lowering/emitter-method-values.ts @@ -40,7 +40,7 @@ export const EMITTER_PROTOTYPE_METHODS: ReadonlySet = new Set([ /** `emit` forwards this many arguments after the event name; a call with * more fences. */ -const EMIT_ARGUMENTS = 16; +const EMIT_ARGUMENTS = 4; /** The runtime's SCR_EE_M_* order: each method's name and the argument * counts its call forwards. Node's methods that look at diff --git a/tests/corpus/6600-emitter-method-values-untyped.ts b/tests/corpus/6600-emitter-method-values-untyped.ts index 1675c97d6..f25be85d9 100644 --- a/tests/corpus/6600-emitter-method-values-untyped.ts +++ b/tests/corpus/6600-emitter-method-values-untyped.ts @@ -40,7 +40,7 @@ console.log(o.removeAllListeners.call(o, "c") === o, o.eventNames()); o.removeAllListeners.call(o); console.log(o.eventNames()); o.on("many", (...args: unknown[]) => console.log("many:", args.length, args.join(","))); -console.log(emit("many", 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11)); +console.log(emit("many", 1, 2, 3, 4)); console.log(o.setMaxListeners.call(o, 3) === o, o.getMaxListeners.call(o)); // off through a detached method value diff --git a/tests/corpus/6601-emitter-method-values.cjs b/tests/corpus/6601-emitter-method-values.cjs index 7873b5f8e..e462bda8e 100644 --- a/tests/corpus/6601-emitter-method-values.cjs +++ b/tests/corpus/6601-emitter-method-values.cjs @@ -38,7 +38,7 @@ const plain = new EventEmitter(); const emit = plain.emit; plain.on("hi", (a, b, c) => console.log("hi", a, b, c)); console.log(emit.call(plain, "hi", 1, "two", [3])); -console.log(emit.call(plain, "hi", 1, 2, 3, 4, 5, 6, 7, 8, 9)); +console.log(emit.call(plain, "hi", 1, 2, 3, 4)); const fn = () => console.log("never"); plain.once("x", fn); const off = plain.off; From 3b4ddd4c8d859d537f46cbe43e14c0e01316ec1f Mon Sep 17 00:00:00 2001 From: Malte Ubl Date: Tue, 20 Oct 2026 18:34:38 -0700 Subject: [PATCH 04/51] Keep object literals' key order through IR rewrites and on shapes typed JSON.parse also builds The key-order witness pass chose a shape's enumeration table from the order its object literals write their keys, but two gaps sent literal values back to the declaration order (JSON.stringify, Object.keys and inspect then differed from Node, silently): - Witness notes are keyed by node identity, and frontend passes that run before the witness pass (the caught-value dispatch, the class dynamic dispatch) rebuild whole function bodies through mapExprChildren, so every record literal in such a function became an unnoted construction that pins its shape. The mapper now reports rebuilt record literals to an observer, and the notes follow them to the rebuilt node. - A shape a checked conversion (typed JSON.parse) also builds stayed pinned to today's table even when the program's own literals build it in another order, such as a message sent to a child and parsed back. It now takes its literals' order; parsed values in another order keep their own through the record-source view, and a shape only checked conversions build keeps today's table. Co-Authored-By: Claude Opus 5.5 --- .../frontend/lowering/key-order-witness.ts | 34 +++++++++--- .../compiler/src/frontend/lowering/lowerer.ts | 6 +++ packages/compiler/src/ir/traverse.ts | 19 ++++++- .../6495-key-order-rewritten-literals.ts.json | 6 +++ .../6496-key-order-built-and-parsed.ts.json | 6 +++ .../6495-key-order-rewritten-literals.ts | 54 +++++++++++++++++++ .../corpus/6496-key-order-built-and-parsed.ts | 35 ++++++++++++ 7 files changed, 150 insertions(+), 10 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6495-key-order-rewritten-literals.ts.json create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6496-key-order-built-and-parsed.ts.json create mode 100644 tests/corpus/6495-key-order-rewritten-literals.ts create mode 100644 tests/corpus/6496-key-order-built-and-parsed.ts diff --git a/packages/compiler/src/frontend/lowering/key-order-witness.ts b/packages/compiler/src/frontend/lowering/key-order-witness.ts index 437a29563..2fc9a6cbd 100644 --- a/packages/compiler/src/frontend/lowering/key-order-witness.ts +++ b/packages/compiler/src/frontend/lowering/key-order-witness.ts @@ -23,10 +23,12 @@ * - any other construction (synthesized builtin results, opaque runtime * producers) pins today's table; * - checked conversions from dynamic values (typed JSON.parse, values - * from JavaScript) pin today's table: the record-source table keeps - * their own key order relative to it, and another table would make - * values parsed in today's order keep a source view. Their order is - * unknown, so a copy out of such a shape cannot be ordered statically; + * from JavaScript) contribute nothing: the record-source table keeps + * their own key order relative to the chosen table. A shape only they + * build keeps today's table, since another table would make values + * parsed in today's order keep a source view; a shape object literals + * also build takes the literals' order. Their order is unknown, so a + * copy out of such a shape cannot be ordered statically; * - a write to an optional key that may be absent appends it in Node. A * key no construction creates ("late-only") goes after every key a * construction can create; any other moving write goes after every key. @@ -56,8 +58,10 @@ export type KeyOrderSegment = | { kind: "opaque" }; /** Witness notes the lowering attaches to the IR nodes it builds. Keyed by - * node identity, so nodes a speculative lowering discards are never walked, - * and a node some later pass rebuilds falls back to pinning. */ + * node identity, so nodes a speculative lowering discards are never walked. + * Literals the IR rewriters rebuild (traverse.ts mapExprChildren) carry + * their notes over (follow); a node rebuilt any other way falls back to + * pinning. */ export class KeyOrderNotes { private readonly literals = new Map(); private readonly copies = new Map(); @@ -77,6 +81,15 @@ export class KeyOrderNotes { return node; } + /** A pass rebuilt `from` as `to` (traverse.ts mapExprChildren): the + * rebuilt literal is the same construction, so it keeps the witness. */ + follow(from: IrExpr, to: IrExpr): void { + const segments = this.literals.get(from); + if (segments !== undefined) this.literals.set(to, segments); + const source = this.copies.get(from); + if (source !== undefined) this.copies.set(to, source); + } + observe(type: IrType): void { if (type.kind === "record") this.observed.add(type.shapeId); } @@ -388,8 +401,13 @@ export function applyKeyOrderWitnesses(lowerer: Lowerer, fns: readonly IrFunctio // A checked conversion builds values in their source's own order, which // the record-source table reproduces relative to the shape's table. A new // table would make every value parsed in today's order keep a source - // view, so such shapes keep today's table. - for (const f of facts.values()) if (f.runtime) f.pinned = true; + // view, so a shape only checked conversions build keeps today's table. + // A shape the program also builds with object literals takes their order: + // the literal's values have no other way to enumerate in it, and values + // parsed from JSON those literals wrote (a message sent to a child or + // another service of the program) arrive in it too. Parsed values in any + // other order keep a source view, exactly as before. + for (const f of facts.values()) if (f.runtime && f.literals.length === 0) f.pinned = true; // Nothing to choose: every witness agrees with today's table and no // optional key is written late. diff --git a/packages/compiler/src/frontend/lowering/lowerer.ts b/packages/compiler/src/frontend/lowering/lowerer.ts index 7addaabf2..932373af2 100644 --- a/packages/compiler/src/frontend/lowering/lowerer.ts +++ b/packages/compiler/src/frontend/lowering/lowerer.ts @@ -30,6 +30,7 @@ import { import { identityPreservingWidening } from "./coercions/identity.js"; import { scanDivergences } from "./divergences.js"; import { applyKeyOrderWitnesses, KeyOrderNotes } from "./key-order-witness.js"; +import { observeRecordLiteralRebuilds } from "../../ir/traverse.js"; import { checkedClassAssertion } from "./class-assertions.js"; import { narrowStoredClassValue, narrowGenericClassValue } from "./class-unions.js"; import { @@ -7371,6 +7372,10 @@ export class Lowerer { * body; both fire edges through the same hooks and are not units * themselves. */ emitReachable(extraRoots?: readonly string[]): { reachable: Set; result: LowerResult } { + // Frontend passes below rewrite function bodies through the IR mapper + // (caught-value and class dispatch); key-order notes follow the record + // literals they rebuild until the witness pass has read them. + observeRecordLiteralRebuilds((from, to) => this.keyOrder.follow(from, to)); this.deferredModuleInitializers = new DeferredModuleInitializers(); const parts = this.splitFiles(); this.narrowing.setFiles(parts.map((fp) => fp.sf)); @@ -7801,6 +7806,7 @@ export class Lowerer { ...this.liftedFns, ...this.implicitFns, ]); + observeRecordLiteralRebuilds(null); for (const finalize of this.shapeOrderHelperFinalizers) finalize(); const functions = [ ...orderedUnits, diff --git a/packages/compiler/src/ir/traverse.ts b/packages/compiler/src/ir/traverse.ts index 3e27cdc36..9db62bce9 100644 --- a/packages/compiler/src/ir/traverse.ts +++ b/packages/compiler/src/ir/traverse.ts @@ -431,6 +431,18 @@ export function everyStmtChild( throw new Error("unhandled IR node"); } +/** Called with each record literal mapExprChildren rebuilds and its + * replacement, so a side table keyed by node identity (the lowering's + * key-order witness notes) follows the literal into the rewritten tree. */ +let recordLiteralRebuilt: ((from: IrExpr, to: IrExpr) => void) | null = null; + +/** Installs (or with null removes) the record-literal rebuild observer. */ +export function observeRecordLiteralRebuilds( + observer: ((from: IrExpr, to: IrExpr) => void) | null, +): void { + recordLiteralRebuilt = observer; +} + /** Copy a node with transformed immediate children; preserve all metadata. */ export function mapExprChildren( node: IrExpr, @@ -619,11 +631,14 @@ export function mapExprChildren( return { ...node, value: expr(node.value) }; case "virtualCall": return { ...node, args: node.args.map((child) => expr(child)) }; - case "recordLit": - return { + case "recordLit": { + const rebuilt: IrExpr = { ...node, fields: node.fields.map((child) => ({ ...child!, value: expr(child.value) })), }; + if (recordLiteralRebuilt !== null) recordLiteralRebuilt(node, rebuilt); + return rebuilt; + } case "recordClone": return { ...node, diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6495-key-order-rewritten-literals.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6495-key-order-rewritten-literals.ts.json new file mode 100644 index 000000000..fef8bfb2a --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6495-key-order-rewritten-literals.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6495-key-order-rewritten-literals.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6496-key-order-built-and-parsed.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6496-key-order-built-and-parsed.ts.json new file mode 100644 index 000000000..24ca2fe96 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6496-key-order-built-and-parsed.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6496-key-order-built-and-parsed.ts" + ], + "diags": [] +} diff --git a/tests/corpus/6495-key-order-rewritten-literals.ts b/tests/corpus/6495-key-order-rewritten-literals.ts new file mode 100644 index 000000000..1cce1db6a --- /dev/null +++ b/tests/corpus/6495-key-order-rewritten-literals.ts @@ -0,0 +1,54 @@ +// Object literals keep their own key order through JSON.stringify when the +// function that builds them is rewritten by a later lowering pass (here a +// catch block that hands the caught value to an `unknown` parameter). The +// declared type lists the keys in another order; Node writes the literal's. +interface StatusBody { + pool?: string; + cpuModel?: string; + cellId?: string; + hiveId?: string; + cluster?: string; + commit?: string; + idleSince?: number; +} +interface PatchBody { + readyState: string; + status: StatusBody; +} + +function describe(value: unknown): string { + return value instanceof Error ? `Error(${value.message})` : String(value); +} + +function send(path: string, body: string): string { + return `${path} ${body}`; +} + +function write(status: StatusBody): string { + return JSON.stringify(status); +} + +function report(commit: string, idleSince: number | undefined, fail: boolean): string[] { + const lines: string[] = []; + try { + if (fail) throw new Error("probe failed"); + } catch (e) { + lines.push(`caught ${describe(e)}`); + } + lines.push( + send( + "/patch", + JSON.stringify({ + readyState: "DOWNLOADING", + status: { cellId: "c1", cluster: "k1", commit, cpuModel: "m1", hiveId: "h1", idleSince }, + } satisfies PatchBody), + ), + ); + lines.push(write({ cellId: "c2", cluster: "k2", commit, cpuModel: "m2", hiveId: "h2" })); + const body = { hiveId: "h3", commit, cellId: "c3" } satisfies StatusBody; + lines.push(send("/status", JSON.stringify(body)), Object.keys(body).join(",")); + return lines; +} + +for (const line of report("abc", 5, true)) console.log(line); +for (const line of report("def", undefined, false)) console.log(line); diff --git a/tests/corpus/6496-key-order-built-and-parsed.ts b/tests/corpus/6496-key-order-built-and-parsed.ts new file mode 100644 index 000000000..278ecd646 --- /dev/null +++ b/tests/corpus/6496-key-order-built-and-parsed.ts @@ -0,0 +1,35 @@ +// A record type the program both builds with an object literal and reads +// back with a typed JSON.parse: the literal enumerates in its own key order +// (not the declaration's), values parsed from the JSON it wrote keep that +// order, and values parsed from JSON in another order keep theirs. +interface Order { + id: string; + region: string; +} +interface Message { + order: Order; + outputs: string[]; + traceCarrier?: Record; + pollMs: number; +} + +function encode(order: Order, outputs: string[], pollMs: number): string { + const message: Message = { outputs, order, pollMs }; + return JSON.stringify(message); +} + +function decode(text: string): Message { + return JSON.parse(text) as Message; +} + +const wire = encode({ region: "iad1", id: "o1" }, ["a", "b"], 250); +console.log(wire); +const back = decode(wire); +console.log(JSON.stringify(back), Object.keys(back).join(","), Object.keys(back.order).join(",")); + +const external = decode('{"pollMs":5,"order":{"id":"o2","region":"sfo1"},"outputs":[],"traceCarrier":{"t":"1"}}'); +console.log(JSON.stringify(external), Object.keys(external).join(",")); + +const declared = decode('{"order":{"id":"o3","region":"fra1"},"outputs":["x"],"pollMs":1}'); +console.log(JSON.stringify(declared), Object.keys(declared).join(",")); +for (const [key, value] of Object.entries(back)) console.log(key, value !== undefined ? JSON.stringify(value) : "-"); From 0cc067430843f259fdcd1834bb4336e6d7121db3 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 08:23:18 -0500 Subject: [PATCH 05/51] Prepare the first build of an executable in a frontend process The frontend, lowering and LLVM emission of an optimized executable build run in a child process when no earlier build of the same invocation emitted its native program object. The parent passes the LLVM text to the native helper as the file the child wrote, so the helper runs after the pages the frontend used are released. Later builds of the invocation find most native partitions cached and peak in the frontend, so they, along with development builds and builds with SCRIPTC_FRONTEND_PROCESS=0, prepare their input in-process. The child runs with the parent's Node options without --eval, --print and their code, or inspector, watch and test runner options. With SCRIPTC_TIMING=1, it reports its phases on the parent's clock. --- packages/cli/test/frontend-process.test.ts | 172 ++++++++++ .../compiler/src/backend/native-codegen.ts | 54 ++- .../src/executable/executable-cache.test.ts | 37 ++ .../src/executable/executable-cache.ts | 33 ++ .../src/frontend-process-args.test.ts | 56 ++++ .../compiler/src/frontend-process-args.ts | 54 +++ packages/compiler/src/frontend-process.ts | 18 + packages/compiler/src/index.ts | 315 ++++++++++++++++-- packages/compiler/src/timing.ts | 17 +- 9 files changed, 714 insertions(+), 42 deletions(-) create mode 100644 packages/cli/test/frontend-process.test.ts create mode 100644 packages/compiler/src/frontend-process-args.test.ts create mode 100644 packages/compiler/src/frontend-process-args.ts create mode 100644 packages/compiler/src/frontend-process.ts diff --git a/packages/cli/test/frontend-process.test.ts b/packages/cli/test/frontend-process.test.ts new file mode 100644 index 000000000..8f7f85e4d --- /dev/null +++ b/packages/cli/test/frontend-process.test.ts @@ -0,0 +1,172 @@ +import { execFile } from "node:child_process"; +import { mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { pathToFileURL } from "node:url"; +import { promisify } from "node:util"; +import { expect, test } from "vitest"; +import { precompiledRuntimePackTarget } from "../../compiler/src/startup-cache.js"; + +// The first optimized executable build of an invocation prepares its input in a frontend process +// (prepareExecutableInputInChild in the compiler's index.ts) when the compiler runs from its built +// files, as the CLI's bootstrap does. Later builds of the invocation, and builds with +// SCRIPTC_FRONTEND_PROCESS=0, prepare it in the compiler's own process. +const execFileAsync = promisify(execFile); +const repoRoot = join(import.meta.dirname, "../../.."); +const bootstrap = join(repoRoot, "packages/cli/dist/bootstrap.js"); +const compilerApi = pathToFileURL(join(repoRoot, "packages/compiler/dist/index.js")).href; +const helperTarget = precompiledRuntimePackTarget(); + +interface Build { + code: number; + stdout: string; + stderr: string; +} + +async function build( + entry: string, + outPath: string, + cacheRoot: string, + inProcess: boolean, +): Promise { + const env: NodeJS.ProcessEnv = { + ...process.env, + SCRIPTC_CACHE_DIR: cacheRoot, + SCRIPTC_TIMING: "1", + }; + delete env["SCRIPTC_NO_CACHE"]; + if (inProcess) env["SCRIPTC_FRONTEND_PROCESS"] = "0"; + else delete env["SCRIPTC_FRONTEND_PROCESS"]; + try { + const { stdout, stderr } = await execFileAsync( + process.execPath, + [bootstrap, "build", entry, "-o", outPath], + { env, maxBuffer: 8 * 1024 * 1024 }, + ); + return { code: 0, stdout, stderr }; + } catch (error) { + const failed = error as { code?: number; stdout?: string; stderr?: string }; + return { code: failed.code ?? 1, stdout: failed.stdout ?? "", stderr: failed.stderr ?? "" }; + } +} + +/** The build's output without its timing lines, whose times differ between runs. */ +const withoutTiming = (text: string): string => + text + .split("\n") + .filter((line) => !line.startsWith("scriptc timing") && !line.startsWith("scriptc lowering")) + .join("\n"); + +test.skipIf(helperTarget === null)( + "the frontend process builds the executable, diagnostics and cache entries the in-process build does", + async () => { + const dir = await mkdtemp(join(tmpdir(), "scriptc-frontend-process-")); + try { + const src = join(dir, "src"); + await mkdir(src); + await writeFile( + join(src, "shapes.ts"), + "export const area = (w: number, h: number): number => w * h;\n", + ); + await writeFile( + join(src, "main.ts"), + 'import { area } from "./shapes.js";\nconsole.log(area(3, 4), [1, 2, 3].map((x) => x * 2).join(","));\n', + ); + const entry = join(src, "main.ts"); + const out = (name: string): string => join(dir, name, "program"); + await mkdir(join(dir, "child")); + await mkdir(join(dir, "inline")); + + const child = await build(entry, out("child"), join(dir, "cache-child"), false); + const inline = await build(entry, out("inline"), join(dir, "cache-inline"), true); + expect(child.code).toBe(0); + expect(inline.code).toBe(0); + expect(child.stderr).toContain('"phase":"frontend-process"'); + expect(inline.stderr).not.toContain('"phase":"frontend-process"'); + expect(await readFile(out("child"))).toEqual(await readFile(out("inline"))); + expect(await readFile(join(dir, "child", "main.ll"), "utf8")).toBe( + await readFile(join(dir, "inline", "main.ll"), "utf8"), + ); + expect((await execFileAsync(out("child"))).stdout).toBe("12 2,4,6\n"); + // The request and result files are removed. + expect( + (await readdir(join(dir, "child"))).filter((name) => name.includes("frontend")), + ).toEqual([]); + + // A repeat finds the executable the frontend process's build published. An edit rebuilds in + // the compiler's own process. + const repeat = await build(entry, out("child"), join(dir, "cache-child"), false); + expect(repeat.code).toBe(0); + expect(repeat.stderr).not.toContain("scriptc lowering"); + expect(repeat.stderr).not.toContain('"phase":"frontend-process"'); + await writeFile( + join(src, "shapes.ts"), + "export const area = (w: number, h: number): number => w * h + 1;\n", + ); + const edited = await build(entry, out("child"), join(dir, "cache-child"), false); + expect(edited.code).toBe(0); + expect(edited.stderr).toContain("scriptc lowering"); + expect(edited.stderr).not.toContain('"phase":"frontend-process"'); + expect((await execFileAsync(out("child"))).stdout).toBe("13 2,4,6\n"); + + // Without the executable cache entries (as for a program whose frontend inputs cannot be + // cached), the rebuild still runs in the compiler's own process: the first build recorded + // its native program object. + await rm(join(dir, "cache-child", "early-exe"), { recursive: true, force: true }); + await rm(join(dir, "cache-child", "early-exe-route"), { recursive: true, force: true }); + const uncached = await build(entry, out("child"), join(dir, "cache-child"), false); + expect(uncached.code).toBe(0); + expect(uncached.stderr).toContain("scriptc lowering"); + expect(uncached.stderr).not.toContain('"phase":"frontend-process"'); + expect((await execFileAsync(out("child"))).stdout).toBe("13 2,4,6\n"); + + // A program the frontend rejects reports the same diagnostics either way. The first build of + // the invocation in an empty cache runs the frontend process. + await writeFile(join(src, "main.ts"), 'const n: number = "not a number";\nconsole.log(n);\n'); + const childError = await build(entry, out("child"), join(dir, "cache-child-error"), false); + const inlineError = await build(entry, out("inline"), join(dir, "cache-inline"), true); + expect(childError.stderr).toContain('"phase":"frontend-process"'); + expect(childError.code).not.toBe(0); + expect(childError.code).toBe(inlineError.code); + expect(withoutTiming(childError.stderr)).toBe(withoutTiming(inlineError.stderr)); + expect(withoutTiming(childError.stderr)).toContain("not a number"); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }, + 240_000, +); + +test.skipIf(helperTarget === null)( + "a compile called from evaluated code runs its frontend process once", + async () => { + const dir = await mkdtemp(join(tmpdir(), "scriptc-frontend-process-eval-")); + try { + const entry = join(dir, "main.ts"); + await writeFile(entry, 'console.log("from eval");\n'); + const { stdout, stderr } = await execFileAsync( + process.execPath, + [ + "--input-type=module", + "--eval", + `import { compile } from ${JSON.stringify(compilerApi)}; + const result = await compile(process.argv[1], { outDir: process.argv[2], outPath: process.argv[3] }); + console.log("compiled", result.ok);`, + entry, + dir, + join(dir, "program"), + ], + { + env: { ...process.env, SCRIPTC_CACHE_DIR: join(dir, "cache"), SCRIPTC_TIMING: "1" }, + maxBuffer: 8 * 1024 * 1024, + }, + ); + expect(stdout).toBe("compiled true\n"); + expect(stderr).toContain('"phase":"frontend-process"'); + expect((await execFileAsync(join(dir, "program"))).stdout).toBe("from eval\n"); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }, + 240_000, +); diff --git a/packages/compiler/src/backend/native-codegen.ts b/packages/compiler/src/backend/native-codegen.ts index 6aab5880a..ff411922a 100644 --- a/packages/compiler/src/backend/native-codegen.ts +++ b/packages/compiler/src/backend/native-codegen.ts @@ -1,8 +1,8 @@ import { execFile } from "node:child_process"; -import { createHash } from "node:crypto"; -import { constants } from "node:fs"; +import { createHash, type Hash } from "node:crypto"; +import { closeSync, constants, openSync, readSync } from "node:fs"; import { createRequire } from "node:module"; -import { access, chmod, mkdir, readFile, rename, rm, stat } from "node:fs/promises"; +import { access, chmod, copyFile, mkdir, readFile, rename, rm, stat } from "node:fs/promises"; import { dirname, join } from "node:path"; import { promisify } from "node:util"; import { @@ -68,9 +68,31 @@ export interface NativeCodegenArtifact { const resolvedHelperCache = new Map>(); +/** LLVM text already written to `file`, of `bytes` bytes, which a build passes on without reading it. */ +export interface LlvmFile { + readonly file: string; + readonly bytes: number; +} + +/** LLVM text as a string, as parts to concatenate, or as a file. */ +export type LlvmInput = string | readonly string[] | LlvmFile; + +export function isLlvmFile(llvm: LlvmInput): llvm is LlvmFile { + return typeof llvm === "object" && !Array.isArray(llvm); +} + +/** The size of `llvm` in bytes. */ +export function llvmInputBytes(llvm: LlvmInput): number { + if (isLlvmFile(llvm)) return llvm.bytes; + return (typeof llvm === "string" ? [llvm] : llvm).reduce( + (bytes, part) => bytes + Buffer.byteLength(part), + 0, + ); +} + export interface NativeCodegenOptions { outputPath: string; - llvm: string | readonly string[]; + llvm: LlvmInput; outputKind: NativeCodegenOutputKind; sourcePath: string; optimization?: "0" | "1" | "2" | "3" | "s" | "z"; @@ -271,14 +293,31 @@ async function resolveHelper( return pending; } +/** Adds the bytes of `file` to `hash`, as `hash.update` of the text written there would. */ +function updateHashFromFile(hash: Hash, file: string): void { + const fd = openSync(file, "r"); + try { + const buffer = Buffer.allocUnsafe(4 * 1024 * 1024); + for (;;) { + const read = readSync(fd, buffer, 0, buffer.length, null); + if (read === 0) break; + hash.update(buffer.subarray(0, read)); + } + } finally { + closeSync(fd); + } +} + function cacheKey( options: NativeCodegenOptions, target: NativeTargetSpec, helper: HelperIdentity, ): string { const hash = createHash("sha256").update("scriptc-native-codegen-v1\0"); - for (const part of typeof options.llvm === "string" ? [options.llvm] : options.llvm) - hash.update(part); + if (isLlvmFile(options.llvm)) updateHashFromFile(hash, options.llvm.file); + else + for (const part of typeof options.llvm === "string" ? [options.llvm] : options.llvm) + hash.update(part); return hash .update("\0") .update(JSON.stringify(target)) @@ -391,7 +430,8 @@ export async function emitNativeArtifact( // concurrent build of another entry that has the same output stem. const input = privateSiblingPath(options.outputPath, "native-input"); try { - await writeLlvmSource(input, options.llvm, 0o600); + if (isLlvmFile(options.llvm)) await copyFile(options.llvm.file, input); + else await writeLlvmSource(input, options.llvm, 0o600); await invoke(helper.binaryPath, [ "emit", "--input", diff --git a/packages/compiler/src/executable/executable-cache.test.ts b/packages/compiler/src/executable/executable-cache.test.ts index ae3c3b7eb..5a633bb45 100644 --- a/packages/compiler/src/executable/executable-cache.test.ts +++ b/packages/compiler/src/executable/executable-cache.test.ts @@ -13,9 +13,11 @@ import { } from "../backend/native-toolchain.js"; import { installDarwinDebugSymbols, readDarwinDebugSymbols } from "../backend/debug-symbols.js"; import { + hasEarlyExecutableNativeBuild, publishEarlyExecutableCache, readEarlyExecutableCache, readRoutedExecutableCache, + recordEarlyExecutableNativeBuild, type EarlyExecutableCacheOptions, type EarlyExecutableNativeFeatures, } from "./executable-cache.js"; @@ -594,3 +596,38 @@ test("the route index still isolates eager and deferred native environments", as )?.executableRestored, ).toBe(true); }); + +test("a recorded native build is found by its invocation", async () => { + const f = await fixture(); + expect(await hasEarlyExecutableNativeBuild(null, f.options)).toBe(false); + expect(await hasEarlyExecutableNativeBuild(f.root, f.options)).toBe(false); + await recordEarlyExecutableNativeBuild(null, f.options); + await recordEarlyExecutableNativeBuild(f.root, f.options); + expect(await hasEarlyExecutableNativeBuild(f.root, f.options)).toBe(true); + // The record does not depend on the compiler implementation or the native environment. + const rebuilt: EarlyExecutableCacheOptions = { + ...f.options, + implementation: "b".repeat(64), + nativeEnvironment: "another driver", + }; + expect(await hasEarlyExecutableNativeBuild(f.root, rebuilt)).toBe(true); + expect( + await hasEarlyExecutableNativeBuild(f.root, { + ...f.options, + outPath: join(f.options.outDir, "other"), + }), + ).toBe(false); + expect(await hasEarlyExecutableNativeBuild(f.root, { ...f.options, sanitize: true })).toBe(false); + // Publishing an executable does not record a native build. + const other = { ...f.options, outPath: join(f.options.outDir, "published") }; + const tracker = new FrontendInputTracker(); + tracker.run(() => trackedReadFile(f.source)); + await publishEarlyExecutableCache(f.root, other, { + llvmPath: f.llvmPath, + irPath: f.irPath, + native, + executableRestored: false, + frontend: tracker.snapshot(), + }); + expect(await hasEarlyExecutableNativeBuild(f.root, other)).toBe(false); +}); diff --git a/packages/compiler/src/executable/executable-cache.ts b/packages/compiler/src/executable/executable-cache.ts index 8f7014dad..1ed23a038 100644 --- a/packages/compiler/src/executable/executable-cache.ts +++ b/packages/compiler/src/executable/executable-cache.ts @@ -327,6 +327,39 @@ function routePath( return join(root, "early-exe-route", routeKey(options)); } +function nativeBuildMarkerPath( + root: string, + options: Omit, +): string { + return join(root, "early-exe-native-build", routeKey(options)); +} + +/** Records that a build of the invocation `options` emitted its native program object, so the + * native helper's partition cache holds its partitions. Unlike the route, the record does not + * depend on the build publishing its executable. */ +export async function recordEarlyExecutableNativeBuild( + root: string | null, + options: Omit, +): Promise { + if (root === null) return; + const marker = nativeBuildMarkerPath(root, options); + await mkdir(dirname(marker), { recursive: true, mode: 0o700 }); + await writeFile(marker, "", { mode: 0o600 }); +} + +/** Whether an earlier build of the invocation `options` recorded its native program object + * (`recordEarlyExecutableNativeBuild`). False when there is no cache. */ +export async function hasEarlyExecutableNativeBuild( + root: string | null, + options: Omit, +): Promise { + if (root === null) return false; + return stat(nativeBuildMarkerPath(root, options)).then( + (info) => info.isFile(), + () => false, + ); +} + function routeIntegrity(stamp: Omit): string { return createHash("sha256") .update("early-executable-route-stamp-v2\0") diff --git a/packages/compiler/src/frontend-process-args.test.ts b/packages/compiler/src/frontend-process-args.test.ts new file mode 100644 index 000000000..a6e9b52e5 --- /dev/null +++ b/packages/compiler/src/frontend-process-args.test.ts @@ -0,0 +1,56 @@ +import { expect, test } from "vitest"; +import { frontendProcessExecArgv } from "./frontend-process-args.js"; + +test("the frontend process keeps heap limits and loaders", () => { + expect( + frontendProcessExecArgv([ + "--max-old-space-size=8192", + "--stack-size=4000", + "--import", + "tsx", + "-r", + "./setup.cjs", + "--conditions=development", + "--enable-source-maps", + ]), + ).toEqual([ + "--max-old-space-size=8192", + "--stack-size=4000", + "--import", + "tsx", + "-r", + "./setup.cjs", + "--conditions=development", + "--enable-source-maps", + ]); +}); + +test("the frontend process drops evaluated code, inspectors, watch and test runner options", () => { + expect( + frontendProcessExecArgv([ + "--max-old-space-size=8192", + "--input-type=module", + "--eval", + "await compile(process.argv[1], {})", + ]), + ).toEqual(["--max-old-space-size=8192"]); + expect(frontendProcessExecArgv(["-e", "code", "-p", "code", "-pe", "code"])).toEqual([]); + expect( + frontendProcessExecArgv(["--eval=code", "--print=code", "--input-type", "module"]), + ).toEqual([]); + expect( + frontendProcessExecArgv([ + "--inspect", + "--inspect-brk=9229", + "--inspect-port", + "9230", + "--watch", + "--watch-path=src", + "--test", + "--test-name-pattern", + "build", + "--test-reporter=spec", + "--expose-gc", + ]), + ).toEqual(["--expose-gc"]); +}); diff --git a/packages/compiler/src/frontend-process-args.ts b/packages/compiler/src/frontend-process-args.ts new file mode 100644 index 000000000..dde9d002f --- /dev/null +++ b/packages/compiler/src/frontend-process-args.ts @@ -0,0 +1,54 @@ +/** Options of the compiler's own process that the frontend process (frontend-process.ts) must not + * receive, with the options among them that take their value as the next argument: the code of + * --eval and --print would run in place of the frontend process's entry, and the inspector, watch + * and test runner options belong to the process that was started with them. */ +const OPTIONS_WITH_VALUES = new Set([ + "-e", + "--eval", + "-p", + "--print", + "--input-type", + "--inspect-port", + "--debug-port", + "--watch-path", + "--test-concurrency", + "--test-coverage-branches", + "--test-coverage-exclude", + "--test-coverage-functions", + "--test-coverage-include", + "--test-coverage-lines", + "--test-global-setup", + "--test-isolation", + "--test-name-pattern", + "--test-reporter", + "--test-reporter-destination", + "--test-shard", + "--test-skip-pattern", + "--test-timeout", +]); +const OPTIONS = new Set(["-i", "--interactive", "-c", "--check", "--test"]); +const OPTION_PREFIXES = ["--inspect", "--debug-port", "--watch", "--test-", "--input-type="]; + +/** The Node.js options of the frontend process: `execArgv` without the options that run code + * (--eval, --print), open an inspector, or watch or test, so that its own entry runs with the + * compiler process's remaining options, such as heap limits and loaders. */ +export function frontendProcessExecArgv(execArgv: readonly string[]): string[] { + const kept: string[] = []; + for (let index = 0; index < execArgv.length; index++) { + const option = execArgv[index]!; + // A short option group such as -pe takes the code that follows it. + if (OPTIONS_WITH_VALUES.has(option) || /^-[a-z]*[ep][a-z]*$/i.test(option)) { + index++; + continue; + } + if ( + OPTIONS.has(option) || + option.startsWith("--eval=") || + option.startsWith("--print=") || + OPTION_PREFIXES.some((prefix) => option.startsWith(prefix)) + ) + continue; + kept.push(option); + } + return kept; +} diff --git a/packages/compiler/src/frontend-process.ts b/packages/compiler/src/frontend-process.ts new file mode 100644 index 000000000..a3d59516f --- /dev/null +++ b/packages/compiler/src/frontend-process.ts @@ -0,0 +1,18 @@ +// The frontend process of an executable build (FrontendProcess and runFrontendProcess in index.ts): +// loads the compiler, then reads the paths of its request and result from stdin. Without them it +// exits: the build found its executable without it. +import { enableCompileCache } from "node:module"; + +try { + enableCompileCache(); +} catch { + // Bytecode caching only speeds up loading. +} +const { runFrontendProcess } = await import("./index.js"); +let input = ""; +for await (const chunk of process.stdin) input += String(chunk); +const [requestPath, resultPath] = input.split("\n"); +if (requestPath !== undefined && requestPath !== "" && resultPath !== undefined) + await runFrontendProcess(requestPath, resultPath); +// The frontend's own helpers (the checker process, timers) must not keep this process alive. +process.exit(0); diff --git a/packages/compiler/src/index.ts b/packages/compiler/src/index.ts index 7dfcb2179..6e9717135 100644 --- a/packages/compiler/src/index.ts +++ b/packages/compiler/src/index.ts @@ -1,5 +1,5 @@ import { optimizationField, type NativeOptimization } from "./backend/optimization.js"; -import { compilationTiming, type CompilationTiming } from "./timing.js"; +import { compilationTiming, type CompilationTiming, type TimingClock } from "./timing.js"; import { prepareExecutableModule } from "./executable/prepare.js"; import { prepareLibrary, @@ -45,8 +45,12 @@ export type { } from "./compile-types.js"; import { compilePackedLibrary } from "./backend/library-pack.js"; import { InternalCompilerError } from "./errors.js"; +import { frontendProcessExecArgv } from "./frontend-process-args.js"; +import { type ChildProcess, spawn } from "node:child_process"; +import { existsSync } from "node:fs"; import { mkdir, readFile, rename, rm, writeFile } from "node:fs/promises"; import { basename, dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; import { clearCcCaches, configuredTargetPlatform, @@ -66,6 +70,8 @@ import { emitLlvmModuleSource, LlvmUnsupportedError } from "./backend/llvm/emitt import { inlineRcOption } from "./backend/llvm/shapes.js"; import { emitNativeArtifact, + llvmInputBytes, + type LlvmInput, NativeCodegenError, writeLlvmSource, } from "./backend/native-codegen.js"; @@ -115,7 +121,11 @@ import { clearResolveCaches } from "./frontend/resolve.js"; import { loadFfiProfile, type FfiProfile } from "./ffi/ffi-manifest.js"; import { executableLinkFeatures } from "./backend/executable-features.js"; import { youngNurseryRequested } from "./backend/young-nursery.js"; -import { FrontendInputTracker, trackedReadFile } from "./frontend/input-tracker.js"; +import { + type FrontendInputSnapshot, + FrontendInputTracker, + trackedReadFile, +} from "./frontend/input-tracker.js"; import { libraryFrontendImplementationFingerprint, publishEarlyLibraryCache, @@ -127,9 +137,11 @@ import { type SemanticLibraryCacheHit, } from "./library/library-cache.js"; import { + hasEarlyExecutableNativeBuild, publishEarlyExecutableCache, publishEarlyExecutableRoute, readEarlyExecutableCache, + recordEarlyExecutableNativeBuild, type EarlyExecutableCacheOptions, type EarlyExecutableNativeFeatures, } from "./executable/executable-cache.js"; @@ -410,10 +422,16 @@ export async function compile( clearCompileSessionCaches(); const frontendInputs = new FrontendInputTracker(); const warnings: CompileWarningSink = { warnings: [], sourceTexts: null }; - const result = await frontendInputs.run(() => - compileTracked(entryPath, opts, frontendInputs, warnings), - ); - return withCompileWarnings(result, warnings); + // compileTracked starts it for a build with no earlier build of the same invocation. + const frontendProcess = preparesInChildProcess(opts) ? new FrontendProcess() : undefined; + try { + const result = await frontendInputs.run(() => + compileTracked(entryPath, opts, frontendInputs, warnings, frontendProcess), + ); + return withCompileWarnings(result, warnings); + } finally { + frontendProcess?.release(); + } } /** Divergence warnings collected while a compile lowers (none when a cache @@ -601,7 +619,7 @@ interface NativeProgramObject { async function emitNativeProgramObject( entryPath: string, opts: CompileRequestOptions, - llvm: string | readonly string[], + llvm: LlvmInput, features: EarlyExecutableNativeFeatures, ): Promise { const stem = basename(entryPath).replace(/\.(ts|mts|cts|js|mjs|cjs)$/, ""); @@ -612,10 +630,7 @@ async function emitNativeProgramObject( const linkPath = `${privateSiblingPath(artifactPath, "native-program-object")}.o`; // The validated program-object artifact is a single object by contract. const target = nativeCodegenTarget(); - const llvmBytes = (typeof llvm === "string" ? [llvm] : llvm).reduce( - (bytes, part) => bytes + Buffer.byteLength(part), - 0, - ); + const llvmBytes = llvmInputBytes(llvm); const partitions = opts.nativeProgramObject === true || target === null ? 1 : nativeProgramPartitions(llvmBytes); // Optimized programs import small runtime functions from the bitcode of @@ -685,7 +700,7 @@ function runtimePackDiagnostic(error: RuntimePackError, entryPath: string): ScrD } interface PreparedExecutable { - llvmSource: string | readonly string[]; + llvmSource: LlvmInput; llvmPath: string; irPath: string | undefined; nativeFeatures: EarlyExecutableNativeFeatures; @@ -840,13 +855,229 @@ async function prepareExecutableInput( return { llvmSource, llvmPath, irPath, nativeFeatures, programSplit, sourceTexts }; } +/** + * The child process that prepares an executable build's input: frontend, lowering and LLVM emission + * (`runFrontendProcess`). Node's heap keeps the pages a build's frontend used for the life of the + * process, so running them in a process of their own leaves the native helper the memory they took. + */ +const frontendProcessEntry = fileURLToPath(new URL("./frontend-process.js", import.meta.url)); + +/** + * Whether `prepareExecutableInputInChild` may prepare this build: optimized executables, when the + * compiler runs from its built files. Development builds split the LLVM text in this process, and + * SCRIPTC_FRONTEND_PROCESS=0 keeps the frontend in-process. + * + * compileTracked starts the process only when no earlier build of the same invocation emitted its + * native program object (`hasEarlyExecutableNativeBuild`). A rebuild finds most native partitions + * in the helper's cache, so its memory peaks in the frontend, and the parent process would add to + * that peak. + */ +function preparesInChildProcess(opts: CompileRequestOptions): boolean { + return ( + process.env["SCRIPTC_FRONTEND_PROCESS"] !== "0" && + (opts.outputKind ?? "exe") === "exe" && + (opts.optimization ?? "release") !== "dev" && + existsSync(frontendProcessEntry) + ); +} + +/** + * The frontend process (frontend-process.ts) of one compile. `start` launches it; it then waits on + * its stdin for the paths of a request and its result (`run`), and exits once its stdin closes + * without them (`release`). + */ +class FrontendProcess { + private child: ChildProcess | undefined; + private exited: Promise | undefined; + private sent = false; + + /** Whether `start` launched the process. */ + get started(): boolean { + return this.child !== undefined; + } + + start(): void { + if (this.child !== undefined) return; + const execArgv = frontendProcessExecArgv(process.execArgv); + const child = spawn(process.execPath, [...execArgv, frontendProcessEntry], { + stdio: ["pipe", "inherit", "inherit"], + }); + this.exited = new Promise((resolveExit) => { + child.on("error", () => resolveExit(null)); + child.on("exit", (code) => resolveExit(code)); + }); + // A failed start surfaces as a failed run; the build then prepares its input in-process. + child.stdin!.on("error", () => undefined); + this.child = child; + } + + /** Sends the request at `requestPath`; resolves with the process's exit code. */ + run(requestPath: string, resultPath: string): Promise { + if (this.child === undefined || this.exited === undefined || this.sent) + return Promise.resolve(null); + this.sent = true; + this.child.stdin!.end(`${requestPath}\n${resultPath}\n`); + return this.exited; + } + + /** Lets the process exit if no request was sent. */ + release(): void { + if (this.child === undefined || this.sent) return; + this.sent = true; + this.child.stdin!.end(); + // The build is done with it: this process need not wait for it to exit. + this.child.unref(); + (this.child.stdin as { unref?: () => void }).unref?.(); + } +} + +interface FrontendProcessRequest { + entryPath: string; + opts: CompileRequestOptions; + buildPlatform: string; + /** The compile's SCRIPTC_TIMING clock, which the frontend process's phases continue. */ + timing: TimingClock; +} + +/** What the frontend process sends back. A thrown error comes back as `failed`. */ +type FrontendProcessResult = + | { + kind: "prepared"; + llvmPath: string; + llvmBytes: number; + irPath: string | undefined; + nativeFeatures: EarlyExecutableNativeFeatures; + sourceTexts: [string, string][]; + warnings: ScrDiagnostic[]; + frontend: FrontendInputSnapshot; + } + | { + kind: "diagnostics"; + diagnostics: ScrDiagnostic[]; + sourceTexts: [string, string][]; + warnings: ScrDiagnostic[]; + frontend: FrontendInputSnapshot; + } + | { kind: "failed" }; + +/** + * `prepareExecutableInput` in the frontend process, with the LLVM text passed on as its file. The + * process's frontend inputs join `frontendInputs` and its warnings `warningSink`. If it fails, the + * input is prepared here, so the build reports the same errors. + */ +async function prepareExecutableInputInChild( + frontendProcess: FrontendProcess, + entryPath: string, + opts: CompileRequestOptions, + ffi: FfiProfile | null, + buildPlatform: string, + timing: CompilationTiming, + timingClock: TimingClock, + frontendInputs: FrontendInputTracker, + warningSink?: CompileWarningSink, +): Promise { + await mkdir(opts.outDir, { recursive: true }); + const requestPath = privateSiblingPath(join(opts.outDir, "frontend-request.json"), "frontend"); + const resultPath = privateSiblingPath(join(opts.outDir, "frontend-result.json"), "frontend"); + let result: FrontendProcessResult = { kind: "failed" }; + try { + const request: FrontendProcessRequest = { entryPath, opts, buildPlatform, timing: timingClock }; + await writeFile(requestPath, JSON.stringify(request), { mode: 0o600 }); + const exitCode = await frontendProcess.run(requestPath, resultPath); + if (exitCode === 0) + result = JSON.parse(await readFile(resultPath, "utf8")) as FrontendProcessResult; + } catch { + result = { kind: "failed" }; + } finally { + await rm(requestPath, { force: true }); + await rm(resultPath, { force: true }); + } + if (result.kind === "failed") + return prepareExecutableInput(entryPath, opts, ffi, buildPlatform, timing, warningSink); + timing("frontend-process"); + for (const probe of result.frontend.probes) frontendInputs.record(probe); + if (!result.frontend.stable) frontendInputs.markUnstable(); + const sourceTexts = new Map(result.sourceTexts); + if (warningSink !== undefined && result.warnings.length > 0) { + warningSink.warnings = result.warnings; + warningSink.sourceTexts = sourceTexts; + } + if (result.kind === "diagnostics") + return { ok: false, diagnostics: result.diagnostics, sourceTexts }; + return { + llvmSource: { file: result.llvmPath, bytes: result.llvmBytes }, + llvmPath: result.llvmPath, + irPath: result.irPath, + nativeFeatures: result.nativeFeatures, + programSplit: null, + sourceTexts, + }; +} + +/** + * The frontend process (frontend-process.ts): prepares the executable input of the request at + * `requestPath` as `compileTracked` would in-process, and writes what it prepared, without the LLVM + * text, to `resultPath`. + */ +export async function runFrontendProcess(requestPath: string, resultPath: string): Promise { + const request = JSON.parse(await readFile(requestPath, "utf8")) as FrontendProcessRequest; + clearCompileSessionCaches(); + const frontendInputs = new FrontendInputTracker(); + const warnings: CompileWarningSink = { warnings: [], sourceTexts: null }; + let result: FrontendProcessResult = { kind: "failed" }; + try { + let ffi: FfiProfile | null = null; + if (request.opts.ffiProfilePath !== undefined) { + const loaded = loadFfiProfile(resolve(request.opts.ffiProfilePath)); + if (!loaded.ok) throw new InternalCompilerError("the FFI profile could not be reloaded"); + ffi = loaded.profile; + } + const prepared = await frontendInputs.run(() => + prepareExecutableInput( + request.entryPath, + request.opts, + ffi, + request.buildPlatform, + compilationTiming(request.timing), + warnings, + ), + ); + const frontend = frontendInputs.snapshot(); + if (!("ok" in prepared)) { + result = { + kind: "prepared", + llvmPath: prepared.llvmPath, + llvmBytes: llvmInputBytes(prepared.llvmSource), + irPath: prepared.irPath, + nativeFeatures: prepared.nativeFeatures, + sourceTexts: [...prepared.sourceTexts], + warnings: warnings.warnings, + frontend, + }; + } else if (!prepared.ok) { + result = { + kind: "diagnostics", + diagnostics: prepared.diagnostics, + sourceTexts: [...prepared.sourceTexts], + warnings: warnings.warnings, + frontend, + }; + } + } catch { + result = { kind: "failed" }; + } + await writeFile(resultPath, JSON.stringify(result), { mode: 0o600 }); +} + async function compileTracked( entryPath: string, opts: CompileRequestOptions, frontendInputs: FrontendInputTracker, warningSink?: CompileWarningSink, + frontendProcess?: FrontendProcess, ): Promise { const timing = compilationTiming(); + const timingStart = Date.now(); entryPath = resolve(entryPath); const outputKind = opts.outputKind ?? "exe"; if ( @@ -1055,16 +1286,10 @@ async function compileTracked( if (outputKind === "exe") { const helperObjectRoute = opts.nativeProgramObject === true || usesPrecompiledRuntimePack(opts, "llvm"); - // Package content and native driver discovery are independent inputs to - // the same cache key. Start both before waiting so edits do not pay their - // filesystem and subprocess latency serially. - const [implementation, nativeEnvironment] = await Promise.all([ - compilerImplementationIdentity(), - helperObjectRoute - ? executableLinkerEnvironmentFingerprint(process.env, nativeCodegenTarget()?.defaultLinker) - : executableNativeEnvironmentFingerprint(), - ]); - earlyCacheOptions = { + const routeOptions: Omit< + EarlyExecutableCacheOptions, + "implementation" | "implementationDependencies" | "nativeEnvironment" + > = { entryPath, outDir: opts.outDir, outPath: opts.outPath, @@ -1094,8 +1319,27 @@ async function compileTracked( ? resolvePlatformLinker(process.env, nativeCodegenTarget()?.defaultLinker) : (process.env["SCRIPTC_CC"] ?? "clang"), ], - nativeEnvironment, nodeVersion: process.version, + }; + // The frontend process prepares the first build of an invocation (preparesInChildProcess). + // Started here, it loads the compiler while this process computes the rest of the cache key. + if ( + frontendProcess !== undefined && + !(await hasEarlyExecutableNativeBuild(cacheRoot, routeOptions)) + ) + frontendProcess.start(); + // Package content and native driver discovery are independent inputs to + // the same cache key. Start both before waiting so edits do not pay their + // filesystem and subprocess latency serially. + const [implementation, nativeEnvironment] = await Promise.all([ + compilerImplementationIdentity(), + helperObjectRoute + ? executableLinkerEnvironmentFingerprint(process.env, nativeCodegenTarget()?.defaultLinker) + : executableNativeEnvironmentFingerprint(), + ]); + earlyCacheOptions = { + ...routeOptions, + nativeEnvironment, implementation: implementation.digest, implementationDependencies: implementation.dependencies, }; @@ -1227,14 +1471,20 @@ async function compileTracked( }; } timing("executable-cache-miss"); - const prepared = await prepareExecutableInput( - entryPath, - opts, - ffi, - buildPlatform, - timing, - warningSink, - ); + const prepared = + frontendProcess?.started === true + ? await prepareExecutableInputInChild( + frontendProcess, + entryPath, + opts, + ffi, + buildPlatform, + timing, + { start: timingStart, previous: Date.now() }, + frontendInputs, + warningSink, + ) + : await prepareExecutableInput(entryPath, opts, ffi, buildPlatform, timing, warningSink); if ("ok" in prepared) return prepared; const { llvmSource, llvmPath, irPath, nativeFeatures, programSplit, sourceTexts } = prepared; const backend = "llvm" as const; @@ -1262,6 +1512,9 @@ async function compileTracked( nativeFeatures, ); timing("native-object"); + await recordEarlyExecutableNativeBuild(cacheRoot, executableCacheOptions).catch( + () => undefined, + ); } catch (err) { if (!(err instanceof NativeCodegenError)) throw err; return { diff --git a/packages/compiler/src/timing.ts b/packages/compiler/src/timing.ts index 79c07cb7e..6d9f6844d 100644 --- a/packages/compiler/src/timing.ts +++ b/packages/compiler/src/timing.ts @@ -1,11 +1,20 @@ export type CompilationTiming = (phase: string, detail?: Record) => void; +/** The start of a timing and its latest phase, as `Date.now()` values. */ +export interface TimingClock { + start: number; + previous: number; +} + /** Opt-in phase durations shared by the Node seed and native compiler. - * The disabled path does not read the clock or write compiler output. */ -export function compilationTiming(): CompilationTiming { + * The disabled path does not read the clock or write compiler output. + * With `clock`, the phases continue a timing that started in another + * process, so a frontend process reports them as its parent compile would. */ +export function compilationTiming(clock?: TimingClock): CompilationTiming { if (process.env["SCRIPTC_TIMING"] !== "1") return () => {}; - const start = performance.now(); - let previous = start; + const opened = performance.now(); + const start = clock === undefined ? opened : opened - (Date.now() - clock.start); + let previous = clock === undefined ? start : opened - (Date.now() - clock.previous); return (phase, detail = {}) => { const now = performance.now(); process.stderr.write( From c1bb553964ddfe547c74932ad5ecef14c622147f Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 08:30:09 -0500 Subject: [PATCH 06/51] Give ProgramLifetime class instances a 32-bit header and packed fields --- .../compiler/src/backend/llvm/classes.test.ts | 72 ++++- packages/compiler/src/backend/llvm/classes.ts | 256 +++++++++++++++--- packages/compiler/src/backend/llvm/emitter.ts | 33 ++- packages/compiler/src/backend/llvm/publish.ts | 82 ++++-- packages/runtime/src/scr_message.c | 2 +- packages/runtime/src/scr_publish.c | 18 +- packages/runtime/src/scr_rc.h | 22 ++ .../4635-contract-program-lifetime-layouts.ts | 224 +++++++++++++++ ...-contract-program-lifetime-layouts.ts.json | 1 + 9 files changed, 625 insertions(+), 85 deletions(-) create mode 100644 tests/corpus/4635-contract-program-lifetime-layouts.ts create mode 100644 tests/harness/corpus-costs/4635-contract-program-lifetime-layouts.ts.json diff --git a/packages/compiler/src/backend/llvm/classes.test.ts b/packages/compiler/src/backend/llvm/classes.test.ts index 8a2a4659a..3c2b0095c 100644 --- a/packages/compiler/src/backend/llvm/classes.test.ts +++ b/packages/compiler/src/backend/llvm/classes.test.ts @@ -7,6 +7,7 @@ import { mangleClassReleaseDirect, mangleClassRelease, mangleClassRetain, + mangleClassStruct, mangleClassTrace, mangleRecordRelease, } from "../mangle.js"; @@ -129,18 +130,19 @@ test("ProgramLifetime hierarchies are never freed and take no reference counts", expect(body(mangleClassRetain(cls)), cls).toMatch(/entry:\n {2}ret ptr %o\n}/); expect(body(mangleClassRelease(cls)), cls).toMatch(/entry:\n {2}ret void\n}/); // new: no collector header (no descriptor, no scr_cyc_alloc) and the - // program-lifetime count 2^31 under the class id (Node's preorder number - // 0 + 1, Leaf's 1 + 1), not SIZE_MAX (which means published). + // 32-bit header: the tag 0x70000000 and the class id (Node's preorder + // number 0 + 1, Leaf's 1 + 1), without a count. const nw = body(mangleClassNew(cls)); const id = cls === "Node" ? 1 : 2; - expect(nw, cls).toContain(`store i64 ${id * 2 ** 32 + 2 ** 31}, ptr %o.vt`); - expect(nw, cls).not.toContain("store i64 -1, ptr %o"); + expect(nw, cls).toContain(`store i32 ${0x70000000 + id}, ptr %o.vt`); + expect(nw, cls).not.toContain("store i64"); expect(nw, cls).not.toContain("scr_cyc_alloc"); expect(llvm, cls).not.toContain(`@${mangleClassTrace(cls)}.cyctype`); // The RC audit (sanitize builds) learns that the program keeps // program-lifetime objects; other builds emit nothing for it. expect(nw, cls).toContain("call void @scr_obj_lifetime_note(ptr %o)"); } + expect(llvm).toContain(`%${mangleClassStruct("Leaf")} = type { i32, ptr }`); const plain = emitLlvmModule(mod, { pointerBits: 64, objectAudit: false }); expect(plain).not.toContain("scr_obj_lifetime_note"); // The mortal Owner keeps its header, counts and possible-root step. @@ -149,6 +151,44 @@ test("ProgramLifetime hierarchies are never freed and take no reference counts", expect(body(mangleClassRelease("Owner"))).toContain("load i32, ptr %bufp"); }); +test("ProgramLifetime classes pack their fields after a 32-bit header", () => { + const node = { kind: "object" as const, className: "Node" }; + const leaf = { kind: "object" as const, className: "Leaf" }; + const BOOL = { kind: "bool" as const }; + const nodeFields = [ + { name: "parent", type: node }, + { name: "flag", type: BOOL }, + { name: "weight", type: F64 }, + ]; + const mod: IrModule = { + irVersion: 17, + sourceFile: loc.file, + entry: "main", + functions: [{ name: "main", params: [], returnType: VOID, locals: [], body: [], loc }], + classes: [ + { name: "Node", fields: nodeFields, methods: [], loc }, + { + name: "Leaf", + base: "Node", + fields: [...nodeFields, { name: "next", type: leaf }, { name: "mark", type: BOOL }], + methods: [], + loc, + }, + ], + contracts: { programLifetime: ["Node"] }, + }; + const llvm = emitLlvmModule(mod, { pointerBits: 64 }); + // Fields go widest first, each at the first offset that fits it. Node's + // weight (an int32 slot: nothing stores a fraction) takes the word the + // header shares (24 bytes, not 32); Leaf keeps Node's offsets and puts its + // bool next to Node's (32 bytes, not 48). + expect(llvm).toContain(`%${mangleClassStruct("Node")} = type { i32, i32, ptr, i8 }`); + expect(llvm).toContain(`%${mangleClassStruct("Leaf")} = type { i32, i32, ptr, i8, i8, ptr }`); + // Without the contract, the layout is the declared one. + const mortal = emitLlvmModule({ ...mod, contracts: {} }, { pointerBits: 64 }); + expect(mortal).toContain(`%${mangleClassStruct("Leaf")} = type { i64, ptr, i8, i32, ptr, i8 }`); +}); + test("published ProgramLifetime instances use the published mark every other object uses", () => { const node = { kind: "object" as const, className: "Node" }; const leaf = { kind: "object" as const, className: "Leaf" }; @@ -212,19 +252,25 @@ test("published ProgramLifetime instances use the published mark every other obj contracts: { programLifetime: ["Node"] }, }; const llvm = emitLlvmModule(mod, { pointerBits: 64 }); - // The walker marks each instance like a mortal object (SIZE_MAX, logged so - // a refusal restores the program-lifetime count); untraced, so it never - // leaves a collector buffer. + // The walker marks each instance by the top bit of its 32-bit header, + // logging the word so a refusal restores it; untraced, so it never leaves + // a collector buffer. expect(llvm).toContain("call void @scr_publish(ptr"); - // Kind 4 (an object) | 0x100 (the rc word keeps a class id: the mark - // writes only the count). - expect(llvm).toMatch(/@scr_pub_mark\(ptr %ctx, ptr %o, i1 false, i32 260\)/); - // The frozen-object guard of the store tests the count against UINT32_MAX: - // an unpublished instance (2^31) stays writable. + // Kind 4 (an object) | 0x200 (the 32-bit header: the mark sets its top + // bit). + expect(llvm).toMatch(/@scr_pub_mark\(ptr %ctx, ptr %o, i1 false, i32 516\)/); + const plain = emitLlvmModule(mod, { pointerBits: 64, objectAudit: false }); + expect(plain).toContain("%mpub = or i32 %mrc, -2147483648"); + // Pushing a reference skips one published before: a set top bit. + expect(llvm).toMatch(/%immortal0 = icmp slt i32 %rc0, 0/); + // The walker dispatches on the class id in the header's low 28 bits. + expect(llvm).toContain("%id = and i32 %idw, 268435455"); + // The frozen-object guard of the store tests the top bit: an unpublished + // instance stays writable. const store = llvm.split(/^define /m).find((fn) => fn.includes("call void @scr_throw_published_field(")) ?? ""; - expect(store).toMatch(/icmp eq i32 %\S+, -1/); + expect(store).toMatch(/icmp slt i32 %\S+, 0/); expect(store).toContain("call void @scr_publish(ptr"); // 32-bit targets use 2^31. const llvm32 = emitLlvmModule(mod, { pointerBits: 32 }); diff --git a/packages/compiler/src/backend/llvm/classes.ts b/packages/compiler/src/backend/llvm/classes.ts index 5ad29430d..061bd06a2 100644 --- a/packages/compiler/src/backend/llvm/classes.ts +++ b/packages/compiler/src/backend/llvm/classes.ts @@ -32,6 +32,7 @@ import { } from "../mangle.js"; import { cycTypeDescriptor, emitObjectAlloc, emitObjectFree } from "./alloc.js"; import { llvmCommentText } from "./common.js"; +import { INLINE_UNION_FIELD } from "./inline-unions.js"; import { NullableRefFields } from "./nullable-fields.js"; import type { Int32Slots } from "../../ir/int32-slots.js"; import { @@ -88,6 +89,15 @@ export class LlClassMeta { * bits, and the hierarchy's vtables are one array indexed by the id * (mangleVtArray). */ idInRc = false; + /** A ProgramLifetime class with the 32-bit header (assignCompactLayouts): + * the i32 at offset 0 is compactHeaderWord, with the class id in place of + * the rc word's high half, and the fields are packed after it. */ + compact = false; + /** Compact classes: the struct's elements by offset (the header, field + * -1, then the fields by their index in def.fields). */ + compactElems: { offset: number; size: number; align: number; type: string; field: number }[] = []; + /** Compact classes: each field's GEP index, by its index in def.fields. */ + compactSlots: number[] = []; constructor(readonly def: IrClassDef) { this.root = this; @@ -371,6 +381,10 @@ export function classSlotWordIndex(meta: LlClassMeta): number { /** The GEP index where a class's own fields start: after the header words * and the slot word. */ export function classFieldBase(meta: LlClassMeta): number { + if (meta.compact) + throw new InternalCompilerError( + `llvm emitter bug: compact class ${meta.def.name} packs its fields`, + ); return headerEnd(meta) + (meta.threadSlot ? 1 : 0); } @@ -387,7 +401,12 @@ export function classFieldIndex(meta: LlClassMeta, field: string): { index: numb throw new InternalCompilerError( `llvm emitter bug: unknown field ${field} on class ${meta.def.name}`, ); - return { index: classFieldBase(meta) + idx, type: meta.def.fields[idx]!.type }; + return { index: classFieldSlot(meta, idx), type: meta.def.fields[idx]!.type }; +} + +/** The GEP index of the field at `idx` in `meta.def.fields`. */ +export function classFieldSlot(meta: LlClassMeta, idx: number): number { + return meta.compact ? meta.compactSlots[idx]! : classFieldBase(meta) + idx; } /** Lines that release the ThreadSlots values of the object `%o` before it @@ -472,8 +491,7 @@ export function classPreLines( ): string[] { if (meta.idInRc) return [ - ` ${pre}.idp = getelementptr inbounds i8, ptr ${obj}, i64 ${CLASS_ID_OFFSET}`, - ` ${pre}.id = load i32, ptr ${pre}.idp`, + ...classIdLoadLines(meta, obj, `${pre}.id`, ""), ` ${pre}.n = add i32 ${pre}.id, -1 ; the class id is the preorder number + 1`, ` ${pre} = zext i32 ${pre}.n to ${sizeType}`, ]; @@ -499,14 +517,33 @@ export function classIdVtLines( const vtt = `%${mangleVtStruct(root.def.name)}`; const n = root.post - root.pre + 1; return [ - ` ${t}idp = getelementptr inbounds i8, ptr ${obj}, i64 ${CLASS_ID_OFFSET}`, - ` ${t}id = load i32, ptr ${t}idp${group}`, + ...classIdLoadLines(meta, `${obj}`, `${t}id`, group), ` ${t}ix = sub i32 ${t}id, ${root.pre + 1}`, ` ${t}ix64 = zext i32 ${t}ix to i64`, ` ${vt} = getelementptr inbounds [${n} x ${vtt}], ptr @${mangleVtArray(root.def.name)}, i64 0, i64 ${t}ix64`, ]; } +/** Lines that load the class id of the LlClassMeta.idInRc instance `obj` + * into the i32 `id`: the high half of the rc word, or the low 28 bits of a + * compact instance's header word (compactHeaderWord). */ +export function classIdLoadLines( + meta: LlClassMeta, + obj: string, + id: string, + group: string, +): string[] { + if (meta.compact) + return [ + ` ${id}.w = load i32, ptr ${obj}${group}`, + ` ${id} = and i32 ${id}.w, ${COMPACT_ID_MASK}`, + ]; + return [ + ` ${id}p = getelementptr inbounds i8, ptr ${obj}, i64 ${CLASS_ID_OFFSET}`, + ` ${id} = load i32, ptr ${id}p${group}`, + ]; +} + /** The byte offset of the class id in the rc word of an instance that * keeps it there (LlClassMeta.idInRc): the high half of the little-endian * 64-bit word. The count is the i32 at offset 0. */ @@ -522,6 +559,124 @@ export function classIdRcWord(meta: LlClassMeta, lifetime: boolean): string { return (BigInt(meta.pre + 1) * 2n ** 32n + (lifetime ? 2n ** 31n : 1n)).toString(); } +/** The 32-bit header of a compact class instance (LlClassMeta.compact): + * COMPACT_HEADER_TAG, the class id (classIdRcWord's preorder number + 1, 0 + * outside a hierarchy) in the low 28 bits, and COMPACT_PUBLISHED once + * publication marks the object. No other object's count reaches + * 0x70000000, so the runtime reads a word of 0xf0000000 and up as published + * for every object (scr_rc.h SCR_RC_PUBLISHED). */ +export const COMPACT_HEADER_TAG = 0x70000000; +export const COMPACT_PUBLISHED = 0x80000000; +export const COMPACT_ID_MASK = 0x0fffffff; + +/** The header word a compact instance is created with. */ +export function compactHeaderWord(meta: LlClassMeta): number { + return COMPACT_HEADER_TAG | (meta.hierarchy ? meta.pre + 1 : 0); +} + +/** Byte size and alignment of a class field's LLVM storage type. */ +function fieldStorage(type: string): { size: number; align: number } { + switch (type) { + case "i8": + return { size: 1, align: 1 }; + case "i32": + return { size: 4, align: 4 }; + case "ptr": + case "double": + return { size: 8, align: 8 }; + case INLINE_UNION_FIELD: + return { size: 16, align: 8 }; + default: + throw new InternalCompilerError(`llvm emitter bug: class field storage ${type}`); + } +} + +/** The LLVM storage type of each of a class's flattened fields. */ +export function classFieldTypes(host: ShapeHost, cls: IrClassDef, int32?: Int32Slots): string[] { + // Whole-program int32 fields (int32-slots.ts) store the exact i32. + return cls.fields.map((f) => + int32?.isField(cls.name, f.name) + ? "i32" + : llFieldType(f.type, host.inlineUnions.classField(cls.name, f.name)), + ); +} + +/** Gives the classes `compact` admits (whole hierarchies: the test is the + * root's) the 32-bit header and packed fields. A class keeps its base's + * field offsets, and places each of its own fields, widest first, at the + * first offset after the header that fits it: in a hole another field's + * alignment left, in the base's tail padding, or at the end. So 32-bit and + * smaller fields share 8-byte words. The struct lists the header and the + * fields by offset, which LLVM's layout of the type reproduces (checked). */ +export function assignCompactLayouts( + metaMap: ReadonlyMap, + compact: (root: LlClassMeta) => boolean, + fieldTypes: (cls: IrClassDef) => string[], +): void { + const byPre = [...metaMap.values()].sort((a, b) => a.pre - b.pre); + for (const meta of byPre) { + if (meta.def.runtime === true || !compact(meta.root)) continue; + const base = meta.base; + const inherited = base?.compactElems ?? [ + { offset: 0, size: 4, align: 4, type: "i32", field: -1 }, + ]; + const baseCount = base?.def.fields.length ?? 0; + if (base !== null && (!base.compact || !sameFieldPrefix(base.def, meta.def))) + throw new InternalCompilerError( + `llvm emitter bug: compact class ${meta.def.name} does not extend its base's layout`, + ); + const elems = inherited.map((e) => ({ ...e })); + const types = fieldTypes(meta.def); + const own = types + .map((type, field) => ({ type, field, ...fieldStorage(type) })) + .slice(baseCount) + .sort((a, b) => b.size - a.size || a.field - b.field); + for (const f of own) { + const sorted = [...elems].sort((a, b) => a.offset - b.offset); + let at = -1; + let cursor = 0; + for (const e of sorted) { + const start = alignTo(cursor, f.align); + if (start + f.size <= e.offset) { + at = start; + break; + } + cursor = Math.max(cursor, e.offset + e.size); + } + if (at < 0) at = alignTo(cursor, f.align); + elems.push({ offset: at, size: f.size, align: f.align, type: f.type, field: f.field }); + } + elems.sort((a, b) => a.offset - b.offset); + // LLVM places each element at the next offset its alignment allows. + let end = 0; + for (const e of elems) { + if (alignTo(end, e.align) !== e.offset) + throw new InternalCompilerError( + `llvm emitter bug: compact layout of ${meta.def.name} is not LLVM's`, + ); + end = e.offset + e.size; + } + const slots = new Array(types.length).fill(-1); + elems.forEach((e, index) => { + if (e.field >= 0) slots[e.field] = index; + }); + meta.compact = true; + meta.compactElems = elems; + meta.compactSlots = slots; + } +} + +function alignTo(n: number, align: number): number { + return Math.ceil(n / align) * align; +} + +function sameFieldPrefix(base: IrClassDef, cls: IrClassDef): boolean { + return ( + base.fields.length <= cls.fields.length && + base.fields.every((f, i) => cls.fields[i]!.name === f.name) + ); +} + function undefFieldInits( host: ClassHost, meta: LlClassMeta, @@ -589,12 +744,17 @@ export function emitClassShapes( for (const cls of emitted) { const meta = metaMap.get(cls.name)!; - // Whole-program int32 fields (int32-slots.ts) store the exact i32. - const fieldTys = cls.fields.map((f) => - int32?.isField(cls.name, f.name) - ? "i32" - : llFieldType(f.type, host.inlineUnions.classField(cls.name, f.name)), - ); + if (meta.compact) { + typeDefs.push( + `%${mangleClassStruct(cls.name)} = type { ${meta.compactElems.map((e) => e.type).join(", ")} } ` + + `; class ${cls.name} (32-bit header) { ${meta.compactElems + .slice(1) + .map((e) => llvmCommentText(cls.fields[e.field]!.name)) + .join("; ")} }`, + ); + continue; + } + const fieldTys = classFieldTypes(host, cls, int32); // Subclasses of a runtime base embed its remaining prefix (the state // and display-name slots) between the vtable word and the field list; // stream subclasses add the stream-state pointer — upcasts to @@ -718,7 +878,7 @@ export function emitClassShapes( const traced = host.tracedShapes.has(`object:${cls.name}`); const runtimePrefix = runtimePrefixOf(meta); const isStreamRooted = streamRooted(meta); - const fieldIndex = (i: number): number => classFieldBase(meta) + i; + const fieldIndex = (i: number): number => classFieldSlot(meta, i); // Teardown, trace and gcFree see each slot's storage type: a nullable // class field holds the instance pointer (NULL for the unit arm), and // every class retain/release/visit entry point is NULL-tolerant. @@ -851,30 +1011,41 @@ export function emitClassShapes( ``, ); // Direct release (the vtable entry): the class's own teardown. The - // caller already handled NULL/immortal. - const reld: string[] = [ - `define internal void @${mangleClassReleaseDirect(cls.name)}(ptr %o) ${FN_ATTRS} { ; direct release ${cls.name}`, - `entry:`, - ` %rc = load ${count}, ptr %o`, - ` %n = sub ${count} %rc, 1`, - ` store ${count} %n, ptr %o`, - ` %dead = icmp eq ${count} %n, 0`, - ` br i1 %dead, label %free, label %${traced ? "root" : "done"}`, - `free:`, - ]; - if (traced) reld.push(...cycleDeadLines(host, "teardown")); - if (bounded) { - const destroy = `${mangleClassReleaseDirect(cls.name)}_destroy`; - const destroyBody: string[] = []; - teardown(destroyBody); - defs.push(...destroyDefinitions(destroy, destroyBody, chained), ``); - host.declare(`declare void @scr_rc_destroy(ptr, ptr)`); - reld.push(` call void @scr_rc_destroy(ptr %o, ptr @${destroy})`); - } else teardown(reld); - reld.push(` br label %done`); - if (traced) reld.push(`root:`, ...cycleRootLines(host, "done")); - reld.push(`done:`, ` ret void`, `}`, ``); - defs.push(...reld); + // caller already handled NULL/immortal. A compact instance has no + // count: nothing releases it. + if (meta.compact) { + defs.push( + `define internal void @${mangleClassReleaseDirect(cls.name)}(ptr %o) ${FN_ATTRS} { ; direct release ${cls.name} (immortal)`, + `entry:`, + ` ret void`, + `}`, + ``, + ); + } else { + const reld: string[] = [ + `define internal void @${mangleClassReleaseDirect(cls.name)}(ptr %o) ${FN_ATTRS} { ; direct release ${cls.name}`, + `entry:`, + ` %rc = load ${count}, ptr %o`, + ` %n = sub ${count} %rc, 1`, + ` store ${count} %n, ptr %o`, + ` %dead = icmp eq ${count} %n, 0`, + ` br i1 %dead, label %free, label %${traced ? "root" : "done"}`, + `free:`, + ]; + if (traced) reld.push(...cycleDeadLines(host, "teardown")); + if (bounded) { + const destroy = `${mangleClassReleaseDirect(cls.name)}_destroy`; + const destroyBody: string[] = []; + teardown(destroyBody); + defs.push(...destroyDefinitions(destroy, destroyBody, chained), ``); + host.declare(`declare void @scr_rc_destroy(ptr, ptr)`); + reld.push(` call void @scr_rc_destroy(ptr %o, ptr @${destroy})`); + } else teardown(reld); + reld.push(` br label %done`); + if (traced) reld.push(`root:`, ...cycleRootLines(host, "done")); + reld.push(`done:`, ` ret void`, `}`, ``); + defs.push(...reld); + } } else { const freeBody: string[] = []; teardown(freeBody); @@ -930,7 +1101,18 @@ export function emitClassShapes( meta.hierarchy && !meta.idInRc ? 16 : 8, ), ); - if (meta.idInRc) { + if (meta.compact && !meta.hierarchy) + nw.push(` store i32 ${compactHeaderWord(meta)}, ptr %o ; no count (compactHeaderWord)`); + else if (meta.compact) { + // The header word: the class id and no count (compactHeaderWord). + // Only publication writes it, setting COMPACT_PUBLISHED, which the + // class id reads mask off, so they keep !invariant.group. + host.declare(`declare ptr @llvm.launder.invariant.group.p0(ptr)`); + nw.push( + ` %o.vt = call ptr @llvm.launder.invariant.group.p0(ptr %o)`, + ` store i32 ${compactHeaderWord(meta)}, ptr %o.vt ; class ${llvmCommentText(cls.name)} is ${meta.pre}`, + ); + } else if (meta.idInRc) { // The class id in the high half of the rc word, the count in the low // half (classIdRcWord). It never changes while the object lives: // retains, releases and the runtime write only the low half. diff --git a/packages/compiler/src/backend/llvm/emitter.ts b/packages/compiler/src/backend/llvm/emitter.ts index 18cee0501..0382efc17 100644 --- a/packages/compiler/src/backend/llvm/emitter.ts +++ b/packages/compiler/src/backend/llvm/emitter.ts @@ -264,7 +264,9 @@ import { emitLibCall, } from "./lib-dispatch.js"; import { + assignCompactLayouts, buildClassGraph, + classFieldTypes, classMembershipIntervals, classFieldIndex, classPreLines, @@ -1492,6 +1494,15 @@ export class LlEmitter { classDefinitions: (cls, sizeOf) => regions.classDefs(cls, sizeOf), }; } + // ProgramLifetime hierarchies count nothing: their instances get the + // 32-bit header and packed fields (classes.ts LlClassMeta.compact), + // except where regions count them or a slot word follows the header. + if (this.sizeType === "i64" && mod.lifetimeRegions !== true) + assignCompactLayouts( + this.classMeta, + (root) => programLifetimeClass(this.shapeHost, root) && !root.threadSlot, + (cls) => classFieldTypes(this.shapeHost, cls, this.int32Slots), + ); for (const [name, rec] of RUNTIME_ERROR_CLASSES) { const meta = this.classMeta.get(name); if (!meta) break; // hand-written IR without the builtin defs: no stamps @@ -5114,6 +5125,7 @@ export class LlEmitter { post: meta.post, runtimeRooted: meta.root.def.runtime === true, idInRc: meta.idInRc, + compact: meta.compact, // Inline scalar-union fields are plain bits: nothing to push. fields: meta.def.fields.flatMap((f) => this.inlineUnions.classField(className, f.name) @@ -5154,15 +5166,21 @@ export class LlEmitter { /** Branches on "`obj` is published" (rc == SIZE_MAX, or a count of * UINT32_MAX in the low half for `count` i32: classes.ts - * LlClassMeta.idInRc); leaves the builder in the cold block and answers - * the continuation label. */ - private emitPublishedTest(obj: string, count: "i32" | "i64" = this.sizeType): string { + * LlClassMeta.idInRc, or the top bit of a compact header word for + * "compact": classes.ts COMPACT_PUBLISHED); leaves the builder in the cold + * block and answers the continuation label. */ + private emitPublishedTest(obj: string, count: "i32" | "i64" | "compact" = this.sizeType): string { const B = this.B; const rc = B.tmp(), frozen = B.tmp(), expected = B.tmp(); - B.line(`${rc} = load ${count}, ptr ${obj}`); - B.line(`${frozen} = icmp eq ${count} ${rc}, -1`); + if (count === "compact") { + B.line(`${rc} = load i32, ptr ${obj}`); + B.line(`${frozen} = icmp slt i32 ${rc}, 0`); + } else { + B.line(`${rc} = load ${count}, ptr ${obj}`); + B.line(`${frozen} = icmp eq ${count} ${rc}, -1`); + } this.declare(`declare i1 @llvm.expect.i1(i1, i1)`); B.line(`${expected} = call i1 @llvm.expect.i1(i1 ${frozen}, i1 false)`); const cold = B.newLabel("pub.frozen"), @@ -5203,7 +5221,10 @@ export class LlEmitter { if (className !== null && !published.classGuarded(className)) return; const B = this.B; const meta = className === null ? undefined : this.classMeta.get(className); - const ok = this.emitPublishedTest(obj, meta?.idInRc === true ? "i32" : this.sizeType); + const ok = this.emitPublishedTest( + obj, + meta?.compact === true ? "compact" : meta?.idInRc === true ? "i32" : this.sizeType, + ); let owner: string; if (meta === undefined) owner = this.cstr("Object"); else if (!meta.hierarchy) owner = this.cstr(meta.def.jsName ?? meta.def.name); diff --git a/packages/compiler/src/backend/llvm/publish.ts b/packages/compiler/src/backend/llvm/publish.ts index 15221bbc3..ebe9b1a64 100644 --- a/packages/compiler/src/backend/llvm/publish.ts +++ b/packages/compiler/src/backend/llvm/publish.ts @@ -25,6 +25,7 @@ import type { IrClassDef, IrRecordShape, IrType, IrUnionDef, SrcLoc } from "../. import { mapOf, STRING, typeKey } from "../../ir/ir.js"; import { mangleClassStruct, mangleRecordStruct } from "../mangle.js"; import { FN_ATTRS } from "./shapes.js"; +import { COMPACT_ID_MASK } from "./classes.js"; import { shapeTagFn } from "./nullable-unions.js"; import type { InlineUnionFields } from "./inline-unions.js"; import { LlvmUnsupportedError } from "./unsupported.js"; @@ -66,6 +67,8 @@ export interface PublishClassInfo { /** Instances keep their class id in the rc word (classes.ts * LlClassMeta.idInRc). */ idInRc: boolean; + /** Instances have the 32-bit header (classes.ts LlClassMeta.compact). */ + compact: boolean; /** The class's own and inherited fields: GEP index and storage type. */ fields: { name: string; index: number; type: IrType }[]; } @@ -76,6 +79,11 @@ const PUB_KIND_UNION = 5; /** SCR_PUB_CLASS_ID, or'ed into a kind: the object keeps its class id * in the high half of its rc word (classes.ts LlClassMeta.idInRc). */ const PUB_CLASS_ID = 0x100; +/** SCR_PUB_COMPACT, or'ed into a kind: the object has the 32-bit header + * (classes.ts compactHeaderWord), whose top bit is the mark. */ +const PUB_COMPACT = 0x200; +/** COMPACT_PUBLISHED as an i32 constant. */ +const COMPACT_MARK = "-2147483648"; /** SCR_PUB_LOG_COUNT (scr_publish.c), in a log entry's `buf`: `rc` is the * count of the low half. */ const PUB_LOG_COUNT = "-2147483648"; @@ -223,7 +231,11 @@ export class PublishWalkers { // hold time-value bits that no walker may push. if (def.arms.some((arm) => scalarRefusal(arm))) throw new LlvmUnsupportedError(`publish of ${typeKey(t)}`, loc); - const live = arms.flatMap((a) => (a.fn === null ? [] : [{ tag: a.tag, fn: a.fn }])); + const live = arms.flatMap((a) => + a.fn === null + ? [] + : [{ tag: a.tag, fn: a.fn, compact: this.compactRef(def.arms[a.tag]!) }], + ); if (this.host.shapeUnion(t)) { // The value is the record itself (or a unit encoding): no box to // mark; the arm's walker marks and walks the record. @@ -256,7 +268,7 @@ export class PublishWalkers { for (const a of live) lines.push( `arm${a.tag}:`, - ...this.push("%slot", a.fn, `u${a.tag}`), + ...this.push("%slot", a.fn, `u${a.tag}`, a.compact), ` br label %done`, ); } @@ -316,18 +328,25 @@ export class PublishWalkers { const lines = [ head, `entry:`, - ...(info.idInRc + ...(info.compact ? [ - ` %idp = getelementptr inbounds i8, ptr %o, i64 4 ; the class id (classes.ts classIdRcWord)`, - ` %id = load i32, ptr %idp`, + ` %idw = load i32, ptr %o ; the class id (classes.ts compactHeaderWord)`, + ` %id = and i32 %idw, ${COMPACT_ID_MASK}`, ` %idpre = add i32 %id, -1`, ` %pre = zext i32 %idpre to ${this.host.sizeType}`, ] - : [ - ` %vtp = getelementptr inbounds ptr, ptr %o, i64 1`, - ` %vt = load ptr, ptr %vtp`, - ` %pre = load ${this.host.sizeType}, ptr %vt`, - ]), + : info.idInRc + ? [ + ` %idp = getelementptr inbounds i8, ptr %o, i64 4 ; the class id (classes.ts classIdRcWord)`, + ` %id = load i32, ptr %idp`, + ` %idpre = add i32 %id, -1`, + ` %pre = zext i32 %idpre to ${this.host.sizeType}`, + ] + : [ + ` %vtp = getelementptr inbounds ptr, ptr %o, i64 1`, + ` %vt = load ptr, ptr %vtp`, + ` %pre = load ${this.host.sizeType}, ptr %vt`, + ]), ` switch ${this.host.sizeType} %pre, label %other [ ${cases.map((c) => `${this.host.sizeType} ${c.pre}, label %c${c.pre}`).join(" ")} ]`, ]; for (const c of cases) @@ -363,6 +382,7 @@ export class PublishWalkers { path, loc, info.idInRc, + info.compact, ); } @@ -374,23 +394,26 @@ export class PublishWalkers { * targets), and an object with a collector header leaves its candidate * buffer through scr_cyc_pub_take. Anything else calls scr_pub_mark, as * does every mark of an RC-audit build. */ - private markLines(cyc: boolean, kind: number, classId = false): string[] { + private markLines(cyc: boolean, kind: number, classId = false, compact = false): string[] { + const flags = compact ? PUB_COMPACT : classId ? PUB_CLASS_ID : 0; const call = (name: string): string[] => [ - ` %${name} = call zeroext i1 @scr_pub_mark(ptr %ctx, ptr %o, i1 ${cyc ? "true" : "false"}, i32 ${classId ? kind | PUB_CLASS_ID : kind})`, + ` %${name} = call zeroext i1 @scr_pub_mark(ptr %ctx, ptr %o, i1 ${cyc ? "true" : "false"}, i32 ${kind | flags})`, ` br i1 %${name}, label %walk, label %done`, ]; if (this.host.rcAudit === true) return call("m"); const S = this.host.sizeType; // An object that keeps its class id in the rc word is marked and // logged through its count, the low half. - const C = classId ? "i32" : S; + // A compact object's header word is logged whole (the restore writes + // the low 32 bits back) and marked by its top bit. + const C = classId || compact ? "i32" : S; const field = (name: string, slot: number): string[] => [ ` %${name}p = getelementptr inbounds ptr, ptr %ctx, i32 ${slot}`, ` %${name} = load ${S === "i64" ? "i64" : "i32"}, ptr %${name}p`, ]; const lines = [ ` %mrc = load ${C}, ptr %o`, - ` %mimm = icmp eq ${C} %mrc, -1`, + compact ? ` %mimm = icmp slt i32 %mrc, 0` : ` %mimm = icmp eq ${C} %mrc, -1`, ` br i1 %mimm, label %done, label %mtest`, `mtest:`, ` %mrefp = getelementptr inbounds ptr, ptr %ctx, i32 6`, @@ -408,7 +431,7 @@ export class PublishWalkers { ok = "%mok2"; } lines.push(` br i1 ${ok}, label %mfast, label %mslow`, `mslow:`, ...call("ms"), `mfast:`); - let buf = classId ? String(PUB_LOG_COUNT) : "0"; + let buf = classId || compact ? String(PUB_LOG_COUNT) : "0"; if (cyc) { // ScrCycHdr.buf, the word before the object: its top bit while the // object is a buffered candidate. @@ -443,7 +466,9 @@ export class PublishWalkers { ` store i32 ${buf}, ptr %mebuf`, ` %mlogged1 = add ${S} %mlogged, 1`, ` store ${S} %mlogged1, ptr %mloggedp`, - ` store ${C} -1, ptr %o`, + ...(compact + ? [` %mpub = or i32 %mrc, ${COMPACT_MARK}`, ` store i32 %mpub, ptr %o`] + : [` store ${C} -1, ptr %o`]), ` br label %walk`, ); return lines; @@ -453,15 +478,18 @@ export class PublishWalkers { * immortal references (the parent links of a tree, values published * before) are the ones scr_pub_push skips: they are tested here, without * the call. The lines end in a fresh block labelled `pushed${id}`. */ - private push(v: string, fn: string, id: string): string[] { + private push(v: string, fn: string, id: string, compact = false): string[] { return [ ` %null${id} = icmp eq ptr ${v}, null`, ` br i1 %null${id}, label %pushed${id}, label %live${id}`, `live${id}:`, // The count (the low half of a 64-bit rc word) is UINT32_MAX for an - // immortal object of every kind (scr_rc.h SCR_RC_COUNT). + // immortal object of every kind (scr_rc.h SCR_RC_COUNT), and a + // compact object's header word has its top bit set once published. ` %rc${id} = load i32, ptr ${v}`, - ` %immortal${id} = icmp eq i32 %rc${id}, -1`, + compact + ? ` %immortal${id} = icmp slt i32 %rc${id}, 0` + : ` %immortal${id} = icmp eq i32 %rc${id}, -1`, ` br i1 %immortal${id}, label %pushed${id}, label %push${id}`, `push${id}:`, ...this.pushLines(v, fn, id), @@ -497,6 +525,12 @@ export class PublishWalkers { ]; } + /** Whether references of type `t` are compact class instances. */ + private compactRef(t: IrType): boolean { + const arm = this.host.nullableArm(t) ?? t; + return arm.kind === "object" && this.host.classInfo(arm.className).compact; + } + private objectBody( head: string, struct: string, @@ -505,8 +539,14 @@ export class PublishWalkers { path: string, loc: SrcLoc, classId = false, + compact = false, ): string[] { - const lines = [head, `entry:`, ...this.markLines(traced, PUB_KIND_OBJECT, classId), `walk:`]; + const lines = [ + head, + `entry:`, + ...this.markLines(traced, PUB_KIND_OBJECT, classId, compact), + `walk:`, + ]; let n = 0; for (const member of members) { const fn = this.walker(member.type, `${path}.${member.name}`, loc); @@ -519,7 +559,7 @@ export class PublishWalkers { lines.push( ` %f${n} = getelementptr inbounds %${struct}, ptr %o, i64 0, i32 ${member.index} ; ${llvmComment(member.name)}`, ` %v${n} = load ptr, ptr %f${n}`, - ...this.push(`%v${n}`, fn, `${n}`), + ...this.push(`%v${n}`, fn, `${n}`, this.compactRef(member.type)), ); n++; } diff --git a/packages/runtime/src/scr_message.c b/packages/runtime/src/scr_message.c index 750dcc97d..cd0e92740 100644 --- a/packages/runtime/src/scr_message.c +++ b/packages/runtime/src/scr_message.c @@ -282,7 +282,7 @@ static size_t scr_message_value(ScrMessageEncoder *encoder, const ScrDyn *input) /* Published values (immortal) travel by pointer: a typed value as a * capsule over the same object, a dyn graph as itself. */ if (input->kind == SCR_DYN_TYPED_REF && input->v.typed_ref.ptr && - SCR_RC_COUNT(input->v.typed_ref.ptr) == SCR_RC_IMMORTAL) { + SCR_RC_PUBLISHED(input->v.typed_ref.ptr)) { scr_sa_thread_shares(); encoder->visits[id].view = (ScrDyn *)input; ScrMessageNode *node = &encoder->message->nodes[id]; diff --git a/packages/runtime/src/scr_publish.c b/packages/runtime/src/scr_publish.c index 0cbb02c8f..218f94d51 100644 --- a/packages/runtime/src/scr_publish.c +++ b/packages/runtime/src/scr_publish.c @@ -42,7 +42,8 @@ typedef struct { } ScrPubLog; /* In ScrPubLog.buf: the object keeps its class id in the high half of - * its rc word (SCR_PUB_CLASS_ID), and `rc` is the count of the low half. */ + * its rc word (SCR_PUB_CLASS_ID), and `rc` is the count of the low half, + * or it is compact (SCR_PUB_COMPACT), and `rc` is its header word. */ #define SCR_PUB_LOG_COUNT UINT32_C(0x80000000) #define SCR_PUB_DYN ((uintptr_t)1) @@ -108,7 +109,7 @@ static void *scr_pub_grow(void *p, size_t *cap, size_t size, void *inline_buf) { void scr_pub_str_fn(void *v, ScrPub *ctx); void scr_pub_push(ScrPub *ctx, void *v, ScrPubFn fn) { - if (!v || SCR_RC_COUNT(v) == SCR_RC_IMMORTAL || ctx->refusal) return; + if (!v || SCR_RC_PUBLISHED(v) || ctx->refusal) return; if (fn == scr_pub_str_fn) { /* a leaf: mark it now */ scr_pub_str_fn(v, ctx); return; @@ -125,9 +126,11 @@ void scr_pub_push(ScrPub *ctx, void *v, ScrPubFn fn) { * touch the objects again. */ bool scr_pub_mark(ScrPub *ctx, void *obj, bool cyc, int kind) { size_t *rc = obj; - if (SCR_RC_COUNT(obj) == SCR_RC_IMMORTAL || ctx->refusal) return false; - bool class_id = (kind & SCR_PUB_CLASS_ID) != 0; - kind &= ~SCR_PUB_CLASS_ID; + if (SCR_RC_PUBLISHED(obj) || ctx->refusal) return false; + bool compact = (kind & SCR_PUB_COMPACT) != 0; + /* Both log the low 32 bits: a count, or a compact header word. */ + bool class_id = compact || (kind & SCR_PUB_CLASS_ID) != 0; + kind &= ~(SCR_PUB_CLASS_ID | SCR_PUB_COMPACT); if (ctx->logged == ctx->log_cap) ctx->log = scr_pub_grow(ctx->log, &ctx->log_cap, sizeof *ctx->log, ctx->log_inline); #ifdef SCR_RC_AUDIT @@ -156,7 +159,8 @@ bool scr_pub_mark(ScrPub *ctx, void *obj, bool cyc, int kind) { } #endif *e = (ScrPubLog){(uintptr_t)obj, logged, buf}; - if (class_id) SCR_RC_COUNT(obj) = SCR_RC_IMMORTAL; + if (compact) SCR_RC_COUNT(obj) |= SCR_RC_COMPACT_PUBLISHED; + else if (class_id) SCR_RC_COUNT(obj) = SCR_RC_IMMORTAL; else *rc = SIZE_MAX; return true; } @@ -382,7 +386,7 @@ static bool scr_pub_keep_ok(size_t bytes) { } void scr_publish(void *root, ScrPubFn fn) { - if (!root || SCR_RC_COUNT(root) == SCR_RC_IMMORTAL) return; + if (!root || SCR_RC_PUBLISHED(root)) return; ScrPub ctx; if (scr_pub_keep.stack) { ctx.stack = scr_pub_keep.stack; diff --git a/packages/runtime/src/scr_rc.h b/packages/runtime/src/scr_rc.h index 889202975..9ce6e385d 100644 --- a/packages/runtime/src/scr_rc.h +++ b/packages/runtime/src/scr_rc.h @@ -27,6 +27,24 @@ typedef size_t ScrRcCount; #define SCR_RC_IMMORTAL ((ScrRcCount)-1) #define SCR_RC_COUNT(obj) (*(ScrRcCount *)(obj)) +/* Compact class instances (64-bit targets; llvm/classes.ts + * LlClassMeta.compact): the instances of a ProgramLifetime hierarchy count + * nothing, so their header is one 32-bit word instead of the rc word, and + * their fields start at offset 4. The word is SCR_RC_COMPACT_TAG, the class + * id (0 outside a hierarchy) in the low 28 bits, and SCR_RC_COMPACT_PUBLISHED + * once publication marks it. No count reaches the tag, so a word read through + * SCR_RC_COUNT tells the two apart, and a count or word of + * SCR_RC_PUBLISHED_MIN and up is published for every object. Only emitted + * code holds compact instances in typed slots; the runtime reaches them + * through their emitted adapters, publication walkers (SCR_PUB_COMPACT) and + * the published test. */ +#define SCR_RC_COMPACT_TAG UINT32_C(0x70000000) +#define SCR_RC_COMPACT_PUBLISHED UINT32_C(0x80000000) +#define SCR_RC_COMPACT_ID_MASK UINT32_C(0x0fffffff) +#define SCR_RC_PUBLISHED_MIN ((ScrRcCount)UINT32_C(0xf0000000)) +/* Whether `obj`, an object of any kind, is published (immortal). */ +#define SCR_RC_PUBLISHED(obj) (SCR_RC_COUNT(obj) >= SCR_RC_PUBLISHED_MIN) + /* The class id in the high half of `obj`'s rc word, or 0 for an object * without one: the high half of any other object's word is 0, or 0xffffffff * (immortal) or 0x80000000 (program-lifetime), and a class id is never one of @@ -636,6 +654,10 @@ enum { * the high half (SCR_RC_COUNT). The mark, and a refusal's restore, write * only the count. */ SCR_PUB_CLASS_ID = 0x100, + /* Or'ed into a kind: a compact class instance (SCR_RC_COMPACT_TAG). The + * mark sets SCR_RC_COMPACT_PUBLISHED; a refusal's restore writes the + * logged word back. */ + SCR_PUB_COMPACT = 0x200, }; void scr_publish(void *root, ScrPubFn fn); void scr_pub_push(ScrPub *ctx, void *v, ScrPubFn fn); diff --git a/tests/corpus/4635-contract-program-lifetime-layouts.ts b/tests/corpus/4635-contract-program-lifetime-layouts.ts new file mode 100644 index 000000000..b62b7fa1a --- /dev/null +++ b/tests/corpus/4635-contract-program-lifetime-layouts.ts @@ -0,0 +1,224 @@ +import { publish } from "@scriptc/threads"; +import type { ProgramLifetime } from "@scriptc/types"; + +// ProgramLifetime classes whose fields mix widths: booleans, small integers, +// numbers, strings, references and optional values, declared in an order +// that leaves alignment holes, with subclasses that add fields of each width. +// Field reads and writes, virtual dispatch, instanceof, boxing into unknown, +// thrown instances and publication behave the same before and after the +// objects are published. + +abstract class Node implements ProgramLifetime { + readonly kind: number; + flagged = false; + readonly pos: number; + end: number; + parent: Node | undefined = undefined; + constructor(kind: number, pos: number, end: number) { + this.kind = kind; + this.pos = pos; + this.end = end; + } + abstract label(): string; + width(): number { + return this.end - this.pos; + } +} + +class Ident extends Node { + readonly text: string; + id = 0; + escaped = false; + constructor(pos: number, text: string) { + super(1, pos, pos + text.length); + this.text = text; + } + label(): string { + return `Ident(${this.text}#${this.id}${this.escaped ? "!" : ""})`; + } +} + +class Literal extends Node { + readonly value: number; + negative: boolean; + radix = 10; + constructor(pos: number, value: number) { + super(2, pos, pos + String(value).length); + this.value = value; + this.negative = value < 0; + } + label(): string { + return `Literal(${this.value.toString(this.radix)}${this.negative ? "-" : ""})`; + } +} + +class Binary extends Node { + readonly left: Node; + readonly right: Node; + operator: string; + precedence = 0; + parenthesized = false; + constructor(left: Node, operator: string, right: Node) { + super(3, left.pos, right.end); + this.left = left; + this.right = right; + this.operator = operator; + left.parent = this; + right.parent = this; + } + label(): string { + const inner = `${this.left.label()} ${this.operator} ${this.right.label()}`; + return this.parenthesized ? `(${inner})` : inner; + } +} + +class Tagged extends Ident { + tag: string | undefined = undefined; + weight = 0.5; + marked = false; + label(): string { + return `Tagged(${super.label()}, ${this.tag ?? "-"}, ${this.weight}, ${this.marked})`; + } +} + +class Flow implements ProgramLifetime { + flags: number; + node: Node | undefined; + antecedent: Flow | undefined; + reachable = true; + constructor(flags: number, node: Node | undefined, antecedent: Flow | undefined) { + this.flags = flags; + this.node = node; + this.antecedent = antecedent; + } +} + +function describe(n: Node): string { + const parent = n.parent === undefined ? "-" : String(n.parent.kind); + return `${n.label()} k${n.kind} ${n.pos}-${n.end} w${n.width()} ${n.flagged ? "F" : "f"} up ${parent}`; +} + +function classify(value: unknown): string { + if (value instanceof Tagged) return `tagged ${value.tag ?? "-"}`; + if (value instanceof Ident) return `ident ${value.text}`; + if (value instanceof Literal) return `literal ${value.value}`; + if (value instanceof Binary) return `binary ${value.operator}`; + if (value instanceof Flow) return `flow ${value.flags}`; + return typeof value; +} + +function isNode(value: unknown): boolean { + return value instanceof Node; +} + +function isLiteral(value: unknown): boolean { + return value instanceof Literal; +} + +function walk(n: Node, out: string[]): void { + out.push(describe(n)); + if (n instanceof Binary) { + walk(n.left, out); + walk(n.right, out); + } +} + +function build(prefix: string): { root: Binary; flows: Flow[]; idents: Ident[] } { + const a = new Ident(0, `${prefix}a`); + a.id = 7; + const b = new Tagged(4, `${prefix}b`); + b.tag = "t"; + b.weight = 1.25; + b.marked = true; + b.escaped = true; + const lit = new Literal(8, -255); + lit.radix = 16; + const sum = new Binary(a, "+", b); + sum.precedence = 12; + sum.parenthesized = true; + const root = new Binary(sum, "*", lit); + root.precedence = 13; + root.flagged = true; + const flows: Flow[] = []; + let prev: Flow | undefined = undefined; + for (const n of [a, b, lit, sum, root]) { + prev = new Flow(flows.length * 3 + 1, n, prev); + flows.push(prev); + } + flows[2]!.reachable = false; + return { root, flows, idents: [a, b] }; +} + +function report(side: string, g: { root: Binary; flows: Flow[]; idents: Ident[] }): void { + const out: string[] = []; + walk(g.root, out); + for (const line of out) console.log(side, line); + for (const f of g.flows) { + const node = f.node === undefined ? "-" : classify(f.node); + const before = f.antecedent === undefined ? "-" : String(f.antecedent.flags); + console.log(side, "flow", f.flags, f.reachable, node, before); + } + const boxed: unknown[] = [g.root, g.root.left, g.idents[1], g.flows[0], 3, "s"]; + console.log(side, boxed.map(classify).join(", ")); +} + +function attempt(what: string, write: () => void): void { + try { + write(); + console.log(what, "wrote"); + } catch (e) { + console.log(what, (e as Error).name, (e as Error).message); + } +} + +const g = build("x"); +report("before", g); + +// Writes to fields that share an 8-byte word with their neighbours. +g.idents[0]!.id = 9; +g.idents[0]!.escaped = true; +const tagged = g.idents[1] as Tagged; +tagged.marked = false; +tagged.weight = -2; +g.root.flagged = false; +g.flows[1]!.flags = 40; +report("written", g); + +// A thrown instance keeps its class. +try { + throw g.idents[1]; +} catch (e) { + console.log("caught", classify(e), isNode(e), isLiteral(e)); +} + +// Instances as map keys and set members. +const seen = new Map(); +const visited = new Set(); +for (const f of g.flows) { + if (f.node !== undefined) seen.set(f.node, (seen.get(f.node) ?? 0) + f.flags); + visited.add(f); +} +console.log("keys", [...seen.entries()].map(([n, v]) => `${n.kind}:${v}`).join(" "), visited.size); + +const published = publish(g); +console.log("same", published === g, publish(g.root) === g.root); +report("published", published); +attempt("ident id", () => { + published.idents[0]!.id = 1; +}); +attempt("tagged flag", () => { + (published.idents[1] as Tagged).marked = true; +}); +attempt("root flag", () => { + published.root.flagged = true; +}); +attempt("flow reachable", () => { + published.flows[2]!.reachable = true; +}); +report("after writes", published); + +// New instances next to published ones stay writable. +const fresh = new Binary(new Ident(16, "y"), "-", new Literal(20, 3)); +fresh.precedence = 11; +fresh.right.flagged = true; +console.log("fresh", describe(fresh), describe(fresh.right), classify(fresh.left), classify(published.root)); diff --git a/tests/harness/corpus-costs/4635-contract-program-lifetime-layouts.ts.json b/tests/harness/corpus-costs/4635-contract-program-lifetime-layouts.ts.json new file mode 100644 index 000000000..2b790bed2 --- /dev/null +++ b/tests/harness/corpus-costs/4635-contract-program-lifetime-layouts.ts.json @@ -0,0 +1 @@ +{ "plain": 4944, "san": 7176 } From 13790b88c75cb33c4588285848d024fc4fb8e9bb Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 08:44:31 -0500 Subject: [PATCH 07/51] Call an object's own or inherited toString in template literals and join --- packages/compiler/src/backend/llvm/dyn.ts | 23 +++++++++-- .../object-literal-tostring-template.ts.json | 6 +++ .../prototype-tostring-conversions.cjs.json | 6 +++ packages/runtime/src/scr_dyn_invoke.c | 40 +++++++++++++------ packages/runtime/src/scr_json.c | 4 +- .../object-literal-tostring-template.ts | 31 ++++++++++++++ .../corpus/prototype-tostring-conversions.cjs | 40 +++++++++++++++++++ .../object-literal-tostring-template.ts.json | 1 + .../prototype-tostring-conversions.cjs.json | 1 + 9 files changed, 135 insertions(+), 17 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/object-literal-tostring-template.ts.json create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/prototype-tostring-conversions.cjs.json create mode 100644 tests/corpus/object-literal-tostring-template.ts create mode 100644 tests/corpus/prototype-tostring-conversions.cjs create mode 100644 tests/harness/corpus-costs/object-literal-tostring-template.ts.json create mode 100644 tests/harness/corpus-costs/prototype-tostring-conversions.cjs.json diff --git a/packages/compiler/src/backend/llvm/dyn.ts b/packages/compiler/src/backend/llvm/dyn.ts index cbaded527..7109fae20 100644 --- a/packages/compiler/src/backend/llvm/dyn.ts +++ b/packages/compiler/src/backend/llvm/dyn.ts @@ -3245,7 +3245,8 @@ export class LlDyn { /* ── the dyn ToString pair (dynToStrHelper, ported) ────────────────── */ /** Node's String() over a dyn value — sc_ds (+1 result) over the - * recursive sc_ds_buf walker. Borrowed operand; never throws. */ + * recursive sc_ds_buf walker. Borrowed operand; an exception that an + * object's toString or valueOf throws stays pending. */ dynToStrHelper(): string { if (this.dynToStrFn) return this.dynToStrFn; const name = "sc_ds"; @@ -3382,7 +3383,8 @@ export class LlDyn { B.startBlock(labels.get(DYN_KIND.OBJ)!); { // The checked-dynamic tree's error encoding renders Error.prototype.toString; - // plain objects are "[object Object]". + // other objects convert with ToPrimitive's string hint: a toString or + // valueOf of their own or of their prototypes, else "[object Object]". const marker = this.objGetLit(B, "%d", "%error"); const isErr = B.tmp(); B.line(`${isErr} = icmp ne ptr ${marker}, null`); @@ -3390,8 +3392,21 @@ export class LlDyn { const lPlain = B.newLabel("ds.pl"); B.condBr(isErr, lErr, lPlain); B.startBlock(lPlain); - this.puts(B, "%b", "[object Object]"); - B.br(done); + { + // Null leaves the exception the conversion threw pending. + host.declare(`declare ptr @scr_dyn_string_coerce_js(ptr)`); + host.declare(`declare void @scr_str_release(ptr)`); + const text = B.tmp(); + B.line(`${text} = call ptr @scr_dyn_string_coerce_js(ptr %d)`); + const converted = B.tmp(); + B.line(`${converted} = icmp ne ptr ${text}, null`); + const lPut = B.newLabel("ds.pp"); + B.condBr(converted, lPut, done); + B.startBlock(lPut); + this.putScrStr(B, "%b", text); + B.line(`call void @scr_str_release(ptr ${text})`); + B.br(done); + } B.startBlock(lErr); // Error.prototype.toString, reading name and message through the // prototype. Null leaves the exception its conversion threw pending. diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/object-literal-tostring-template.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/object-literal-tostring-template.ts.json new file mode 100644 index 000000000..f2c993f6a --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/object-literal-tostring-template.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/object-literal-tostring-template.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/prototype-tostring-conversions.cjs.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/prototype-tostring-conversions.cjs.json new file mode 100644 index 000000000..dcb13023c --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/prototype-tostring-conversions.cjs.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/prototype-tostring-conversions.cjs" + ], + "diags": [] +} diff --git a/packages/runtime/src/scr_dyn_invoke.c b/packages/runtime/src/scr_dyn_invoke.c index 782fab829..92dd944ee 100644 --- a/packages/runtime/src/scr_dyn_invoke.c +++ b/packages/runtime/src/scr_dyn_invoke.c @@ -42,9 +42,12 @@ * the emitted sc_ds walker exists only in programs that spell * String(unknown) themselves). Same rules: arrays join with "," and * null/undefined elements print empty, the checked-dynamic tree's error encoding renders - * "name: message", functions render the native-code form, plain objects - * are [object Object]. */ -static void scr_dyn_display_buf(ScrJsonBuf *b, const ScrDyn *d) { + * "name: message", functions render the native-code form. With + * `to_string` (join and the default sort order, which run ToString), other + * objects convert with ToPrimitive's string hint, calling a toString or + * valueOf of their own or of their prototypes, and an exception it throws + * stays pending; without it (error messages), they are [object Object]. */ +static void scr_dyn_display_buf(ScrJsonBuf *b, const ScrDyn *d, bool to_string) { switch (d->kind) { case SCR_DYN_SYMBOL: { ScrStr *s = scr_dyn_string_coerce(d); @@ -74,7 +77,7 @@ static void scr_dyn_display_buf(ScrJsonBuf *b, const ScrDyn *d) { if (i > 0) scr_jb_putc(b, ','); ScrDyn *e = scr_dyn_arr_at(d, (double)i); if (!e) return; - if (e->kind != SCR_DYN_UNDEF && e->kind != SCR_DYN_NULL) scr_dyn_display_buf(b, e); + if (e->kind != SCR_DYN_UNDEF && e->kind != SCR_DYN_NULL) scr_dyn_display_buf(b, e, to_string); scr_dyn_release(e); if (scr_exc_pending()) return; } @@ -88,6 +91,13 @@ static void scr_dyn_display_buf(ScrJsonBuf *b, const ScrDyn *d) { scr_str_release(text); return; } + if (to_string) { + ScrStr *text = scr_dyn_string_coerce_js(d); + if (!text) return; + scr_jb_write(b, text->data, text->len); + scr_str_release(text); + return; + } scr_jb_puts(b, "[object Object]"); return; } @@ -129,7 +139,7 @@ static void scr_dyn_display_buf(ScrJsonBuf *b, const ScrDyn *d) { case SCR_DYN_TYPED_REF: { ScrDyn *materialized = scr_dyn_typed_ref_materialize(d); - scr_dyn_display_buf(b, materialized); + scr_dyn_display_buf(b, materialized, to_string); scr_dyn_release(materialized); return; } @@ -142,7 +152,7 @@ static void scr_dyn_display_buf(ScrJsonBuf *b, const ScrDyn *d) { ScrStr *scr_dyn_display(const ScrDyn *d) { ScrJsonBuf b; scr_jb_init(&b); - scr_dyn_display_buf(&b, d); + scr_dyn_display_buf(&b, d, false); return scr_jb_finish(&b); } @@ -217,7 +227,7 @@ static bool dyn_cb_check(ScrDyn *const *args, size_t argc) { ScrJsonBuf b; scr_jb_init(&b); - scr_dyn_display_buf(&b, cb); + scr_dyn_display_buf(&b, cb, false); scr_jb_puts(&b, " is not a function"); scr_throw_error(SCR_ERR_TYPE, scr_jb_finish(&b)); return false; @@ -564,7 +574,7 @@ static ScrDyn *scr_dynh_dispatch(ScrDyn *recv, const char *method, ScrDyn *const static ScrStr *dyn_sort_str(const ScrDyn *e) { ScrJsonBuf b; scr_jb_init(&b); - scr_dyn_display_buf(&b, e); + scr_dyn_display_buf(&b, e, true); return scr_jb_finish(&b); } static bool dyn_sort_less(void *ctx, uint64_t xs, uint64_t ys, bool *failed) { @@ -581,7 +591,13 @@ static bool dyn_sort_less(void *ctx, uint64_t xs, uint64_t ys, bool *failed) { return v < 0; } ScrStr *xstr = dyn_sort_str(x); - ScrStr *ystr = dyn_sort_str(y); + ScrStr *ystr = scr_exc_pending() ? NULL : dyn_sort_str(y); + if (scr_exc_pending()) { + scr_str_release(xstr); + if (ystr) scr_str_release(ystr); + *failed = true; + return false; + } int c = scr_str_cmp_u16(xstr, ystr); scr_str_release(xstr); scr_str_release(ystr); @@ -1199,7 +1215,7 @@ static ScrDyn *scr_dyn_invoke_impl( if (i > 0) scr_jb_write(&b, separator->data, separator->len); ScrDyn *e = scr_dyn_arr_at(recv, (double)i); if (e) { - if (e->kind != SCR_DYN_UNDEF && e->kind != SCR_DYN_NULL) scr_dyn_display_buf(&b, e); + if (e->kind != SCR_DYN_UNDEF && e->kind != SCR_DYN_NULL) scr_dyn_display_buf(&b, e, true); scr_dyn_release(e); } if (!e || scr_exc_pending()) { scr_str_release(separator); free(b.data); return NULL; } @@ -1463,7 +1479,7 @@ static ScrDyn *scr_dyn_invoke_impl( ScrJsonBuf b; scr_jb_init(&b); scr_jb_puts(&b, "The comparison function must be either a function or undefined: "); - scr_dyn_display_buf(&b, cmp); + scr_dyn_display_buf(&b, cmp, false); scr_throw_error(SCR_ERR_TYPE, scr_jb_finish(&b)); return NULL; } @@ -2181,7 +2197,7 @@ static ScrDyn *scr_dyn_define_props_internal(ScrDyn *target, ScrDyn *descs, ScrD ScrJsonBuf b; scr_jb_init(&b); scr_jb_puts(&b, "Property description must be an object: "); - scr_dyn_display_buf(&b, descriptor); + scr_dyn_display_buf(&b, descriptor, false); scr_throw_error(SCR_ERR_TYPE, scr_jb_finish(&b)); scr_dyn_release(descriptor); goto fail; diff --git a/packages/runtime/src/scr_json.c b/packages/runtime/src/scr_json.c index 056d67c89..0281e5ca6 100644 --- a/packages/runtime/src/scr_json.c +++ b/packages/runtime/src/scr_json.c @@ -7503,7 +7503,9 @@ ScrStr *scr_dyn_to_string(const ScrDyn *d, const ScrStr *enc) { } const ScrDyn *e = d->v.arr.items[i]; if (e->kind == SCR_DYN_UNDEF || e->kind == SCR_DYN_NULL) continue; /* JS join: empty */ - ScrStr *piece = scr_dyn_string_coerce(e); + /* ToString with the object protocol: an element's own or inherited + * toString/valueOf runs, as in Array.prototype.join. */ + ScrStr *piece = scr_dyn_string_coerce_js(e); if (!piece || scr_exc_pending()) { scr_str_release(piece); scr_str_release(out); diff --git a/tests/corpus/object-literal-tostring-template.ts b/tests/corpus/object-literal-tostring-template.ts new file mode 100644 index 000000000..f87d407f8 --- /dev/null +++ b/tests/corpus/object-literal-tostring-template.ts @@ -0,0 +1,31 @@ +// Template literals and String() call the toString of class instances and +// of object literals that define one, and print "[object Object]" for +// objects that inherit Object.prototype.toString. +class Point { + x: number; + constructor(x: number) { + this.x = x; + } + toString(): string { + return `Point(${this.x})`; + } +} +class Bare { + y: number; + constructor(y: number) { + this.y = y; + } +} +const labeled = { + v: 1, + toString() { + return "labeled" + this.v; + }, +}; +const plain = { v: 2 }; +const p = new Point(3); +const b = new Bare(4); +console.log(`${p}`, `${b}`, `${labeled}`, `${plain}`); +console.log(String(p), String(labeled), p + "", labeled + "", "" + b); +const maybe: Point | undefined = p; +console.log(`${maybe}`); diff --git a/tests/corpus/prototype-tostring-conversions.cjs b/tests/corpus/prototype-tostring-conversions.cjs new file mode 100644 index 000000000..b4983893f --- /dev/null +++ b/tests/corpus/prototype-tostring-conversions.cjs @@ -0,0 +1,40 @@ +'use strict'; +// ToString of objects whose toString or valueOf comes from the object or +// its prototype chain: template literals, join, String(), concatenation, +// Array.prototype.toString and the default sort order call it, as in Node. +function RefNum(num) { this.num = num; } +RefNum.prototype.toString = function () { return "R" + this.num; }; +function Inherit() {} +Inherit.prototype = Object.create(RefNum.prototype); +Inherit.prototype.num = 9; + +const r = new RefNum(5); +const deep = new Inherit(); +const own = { toString() { return "own"; } }; +const viaValueOf = { valueOf() { return 42; } }; +const both = { toString() { return "ts"; }, valueOf() { return 7; } }; +const plain = {}; +const m = new Map([[1, r]]); + +console.log(`${r}`, `${deep}`, `${own}`, `${viaValueOf}`, `${both}`, `${plain}`); +console.log(`${m.get(1) || 0}`, `${m.get(2) || 0}`, r * 2); +console.log([r, deep, own, both, plain].join("|")); +console.log([r, own].toString(), String([r, own]), `${[r, own]}`); +console.log([[r, [own]], null, undefined, deep].join(";")); +console.log(String(r), r + "", own + "", both + "", viaValueOf + "", "x".concat(r)); +console.log([new RefNum(3), new RefNum(1), own, new RefNum(2)].sort().join(",")); + +// A toString returning an object falls back to valueOf; a null prototype +// leaves neither, which throws. +const objectResult = { toString() { return {}; }, valueOf() { return "vo"; } }; +console.log(`${objectResult}`, [objectResult].join()); +const bare = Object.create(null); +try { console.log(`${bare}`); } catch (e) { console.log(e.name, e.message); } +try { console.log([bare].join()); } catch (e) { console.log(e.name, e.message); } + +// An exception from toString propagates out of each conversion. +const thrower = { toString() { throw new Error("boom"); } }; +try { console.log(`${thrower}`); } catch (e) { console.log("template", e.message); } +try { console.log([thrower].join()); } catch (e) { console.log("join", e.message); } +try { console.log(String([thrower])); } catch (e) { console.log("String", e.message); } +try { console.log([new RefNum(1), thrower].sort()); } catch (e) { console.log("sort", e.message); } diff --git a/tests/harness/corpus-costs/object-literal-tostring-template.ts.json b/tests/harness/corpus-costs/object-literal-tostring-template.ts.json new file mode 100644 index 000000000..d9a03264c --- /dev/null +++ b/tests/harness/corpus-costs/object-literal-tostring-template.ts.json @@ -0,0 +1 @@ +{ "plain": 3551, "san": 5414 } diff --git a/tests/harness/corpus-costs/prototype-tostring-conversions.cjs.json b/tests/harness/corpus-costs/prototype-tostring-conversions.cjs.json new file mode 100644 index 000000000..1a9f924ff --- /dev/null +++ b/tests/harness/corpus-costs/prototype-tostring-conversions.cjs.json @@ -0,0 +1 @@ +{ "plain": 4028, "san": 4282 } From 726c2ce3d7466f318f3fde4200c5a216965e1314 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 08:47:37 -0500 Subject: [PATCH 08/51] Freeze, seal and query functions through their property tables A plain function's own properties (name, length, prototype, and the ones a program assigns, as in `hash.outputLen = 32`) live in its property table. Object.freeze, Object.seal and Object.preventExtensions of a function now act on that table and answer the function, and Object.isFrozen, Object.isSealed and Object.isExtensible answer from it, as in Node: - Object.freeze of a function was refused as a possibly-aliased value, and Object.isFrozen of one threw SC2020; - Object.isExtensible answered false and Object.isSealed true for every function. Adding a property to a frozen or non-extensible function throws Node's TypeError; assigning a read-only one throws the runtime's existing read-only TypeError. Class constructors keep their fences. This lets @noble/hashes' createHasher return its frozen hasher, so the sha256 program in the issue prints the digest Node prints. Fixes #1437 Co-authored-by: saboursepideh <25549001+saboursepideh@users.noreply.github.com> --- .../builtin-apis/object-freezing.mdx | 3 ++ .../src/frontend/lowering/lower-calls.ts | 47 +++++++++++++++++++ .../tests/corpus/function-integrity.js.json | 6 +++ packages/runtime/src/scr_json.c | 44 +++++++++++++++++ tests/corpus/function-integrity.js | 44 +++++++++++++++++ 5 files changed, 144 insertions(+) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/function-integrity.js.json create mode 100644 tests/corpus/function-integrity.js diff --git a/docs/content/limitations/builtin-apis/object-freezing.mdx b/docs/content/limitations/builtin-apis/object-freezing.mdx index 38f0155c0..99b78ded5 100644 --- a/docs/content/limitations/builtin-apis/object-freezing.mdx +++ b/docs/content/limitations/builtin-apis/object-freezing.mdx @@ -5,4 +5,7 @@ A write to a read-only field throws Node's `TypeError` in strict mode code and leaves the field unchanged in sloppy mode code. `Object.defineProperty` with a string-literal key changes the value, `writable`, `enumerable`, and `configurable` attributes of a declared field; an object-literal accessor descriptor (`get` or `set`) for a declared field is refused at compile time with `SC2020`, and another accessor descriptor for a declared field throws `SC2020` when the call runs. Freezing, sealing, or preventing extensions of a class instance held in an untyped value throws `SC2020`. + Functions support `Object.freeze`, `Object.seal`, `Object.preventExtensions`, `Object.isFrozen`, `Object.isSealed`, and `Object.isExtensible`; the operations apply to the function's own properties (`name`, `length`, `prototype`, and the ones a program assigns). + Class constructors are not supported. + A write to a read-only property of a checked dictionary or a function throws a `TypeError` whose message omits Node's `'x' of object '#'` (or `of function '…'`) suffix. A JavaScript class field that is first created after its instance became non-extensible is added instead of throwing. diff --git a/packages/compiler/src/frontend/lowering/lower-calls.ts b/packages/compiler/src/frontend/lowering/lower-calls.ts index d768cec9b..581cf0f5c 100644 --- a/packages/compiler/src/frontend/lowering/lower-calls.ts +++ b/packages/compiler/src/frontend/lowering/lower-calls.ts @@ -8071,6 +8071,8 @@ function lowerObjectStaticCall( loc, }; } + const fnIntegrity = functionIntegrity(lowerer, member, value, locOf(call)); + if (fnIntegrity) return fnIntegrity; if (member === "preventExtensions" || member === "seal") { const integrity = lowerClassIntegrity(lowerer, call, member, value); if (integrity) return integrity; @@ -8154,6 +8156,8 @@ function lowerObjectStaticCall( ) ) return value; + const fnIntegrity = functionIntegrity(lowerer, "freeze", value, locOf(call)); + if (fnIntegrity) return fnIntegrity; const integrity = lowerClassIntegrity(lowerer, call, "freeze", value); if (integrity) return integrity; lowerer.noLowering( @@ -10639,3 +10643,46 @@ function lowerObjectAssignOntoError(lowerer: Lowerer, call: ts.CallExpression): loc, }; } + +/** `Object.freeze(fn)` and the other integrity operations on a function + * value: the function's checked-dynamic face holds its own properties (the + * ones a program assigns, `name`, `length`, `prototype`) and their + * integrity state, which the runtime keeps on its property table. The + * mutators answer the function itself; the queries answer from the table. + * Null for other values. */ +function functionIntegrity( + lowerer: Lowerer, + member: string, + value: IrExpr, + loc: SrcLoc, +): IrExpr | null { + if (value.type.kind !== "func" || !lowerer.dynConvertible(value.type)) return null; + const operations = { + freeze: "dyn.freeze", + seal: "dyn.seal", + preventExtensions: "dyn.preventExtensions", + isSealed: "dyn.isSealed", + isExtensible: "dyn.isExtensible", + } as const; + if (!(member in operations)) return null; + const fn = operations[member as keyof typeof operations]; + if (fn === "dyn.isSealed" || fn === "dyn.isExtensible") { + return { kind: "libCall", fn, args: [lowerer.coerceToExpected(value, DYN)], type: BOOL, loc }; + } + const held = lowerer.declareHiddenLocal("%integrityFn", value.type); + const ref = (): IrExpr => ({ kind: "varRef", localId: held.id, type: value.type, loc }); + return { + kind: "seqExpr", + stmts: [ + { kind: "varDecl", localId: held.id, init: value, loc }, + { + kind: "exprStmt", + expr: { kind: "libCall", fn, args: [lowerer.coerceToExpected(ref(), DYN)], type: DYN, loc }, + loc, + }, + ], + result: ref(), + type: value.type, + loc, + }; +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/function-integrity.js.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/function-integrity.js.json new file mode 100644 index 000000000..0f4628189 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/function-integrity.js.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/function-integrity.js" + ], + "diags": [] +} diff --git a/packages/runtime/src/scr_json.c b/packages/runtime/src/scr_json.c index 056d67c89..f19bc7083 100644 --- a/packages/runtime/src/scr_json.c +++ b/packages/runtime/src/scr_json.c @@ -1990,10 +1990,25 @@ static bool scr_dyn_is_extensible_view(const ScrDyn *view) { return (view->kind == SCR_DYN_OBJ || view->kind == SCR_DYN_ARR) && !view->non_extensible; } +/* A plain function's own properties (name, length, prototype, and the ones + * a program adds) live in its property table, which carries the function's + * integrity state too: freezing or sealing the function freezes or seals + * the table. +1, or NULL for other values and for class constructors, whose + * statics keep their own fences. */ +static ScrDyn *scr_dyn_fn_integrity_table(const ScrDyn *object) { + if (object->kind != SCR_DYN_FUNC || object->v.fn.class_obj) return NULL; + return scr_dyn_fn_properties(object); +} + ScrDyn *scr_dyn_prevent_extensions(ScrDyn *object) { if (scr_dyn_class_reflection_fence(object)) return NULL; if (object->kind == SCR_DYN_PROXY) { scr_dyn_proxy_unsupported("Object.preventExtensions"); return NULL; } if (scr_dyn_class_integrity_fence(object, "Object.preventExtensions")) return NULL; + ScrDyn *table = scr_dyn_fn_integrity_table(object); + if (table) { + table->non_extensible = true; + scr_dyn_release(table); + } if (object->kind == SCR_DYN_OBJ || object->kind == SCR_DYN_ARR) { object->non_extensible = true; if (object->kind == SCR_DYN_ARR && object->v.arr.properties) @@ -2007,6 +2022,12 @@ bool scr_dyn_is_extensible(ScrDyn *object) { if (object->kind == SCR_DYN_PROXY) { scr_dyn_proxy_unsupported("Object.isExtensible"); return false; } bool answer; if (scr_dyn_typed_ref_view_answer(object, scr_dyn_is_extensible_view, &answer)) return answer; + ScrDyn *table = scr_dyn_fn_integrity_table(object); + if (table) { + answer = !table->non_extensible; + scr_dyn_release(table); + return answer; + } return scr_dyn_is_extensible_view(object); } @@ -2015,6 +2036,11 @@ ScrDyn *scr_dyn_seal(ScrDyn *object) { if (object->kind == SCR_DYN_PROXY) { scr_dyn_proxy_unsupported("Object.seal"); return NULL; } if (scr_dyn_class_integrity_fence(object, "Object.seal")) return NULL; if (object->symbol_properties) scr_dyn_release(scr_dyn_seal(object->symbol_properties)); + ScrDyn *table = scr_dyn_fn_integrity_table(object); + if (table) { + scr_dyn_release(scr_dyn_seal(table)); + scr_dyn_release(table); + } if (object->kind == SCR_DYN_OBJ) { object->non_extensible = true; for (size_t i = 0; i < object->v.obj.len; i++) object->v.obj.entries[i].configurable = false; @@ -2033,6 +2059,12 @@ bool scr_dyn_is_sealed(const ScrDyn *object) { bool answer; if (scr_dyn_typed_ref_view_answer((ScrDyn *)object, scr_dyn_is_sealed, &answer)) return answer; if (object->symbol_properties && !scr_dyn_is_sealed(object->symbol_properties)) return false; + ScrDyn *table = scr_dyn_fn_integrity_table(object); + if (table) { + answer = scr_dyn_is_sealed(table); + scr_dyn_release(table); + return answer; + } if (object->kind == SCR_DYN_ARR) return object->non_extensible && (object->v.arr.len == 0 || object->v.arr.sealed) && (!object->v.arr.properties || scr_dyn_is_sealed(object->v.arr.properties)); @@ -14383,6 +14415,12 @@ ScrDyn *scr_dyn_freeze(ScrDyn *value) { if (scr_dyn_class_reflection_fence(value)) return NULL; if (scr_dyn_class_integrity_fence(value, "Object.freeze")) return NULL; if (value->symbol_properties) scr_dyn_release(scr_dyn_freeze(value->symbol_properties)); + ScrDyn *table = scr_dyn_fn_integrity_table(value); + if (table) { + scr_dyn_release(scr_dyn_freeze(table)); + scr_dyn_release(table); + return scr_dyn_retain(value); + } if (value->kind == SCR_DYN_ARR) { value->non_extensible = true; value->v.arr.sealed = true; @@ -14412,6 +14450,12 @@ bool scr_dyn_is_frozen(const ScrDyn *value) { bool answer; if (scr_dyn_typed_ref_view_answer((ScrDyn *)value, scr_dyn_is_frozen, &answer)) return answer; if (value->symbol_properties && !scr_dyn_is_frozen(value->symbol_properties)) return false; + ScrDyn *table = scr_dyn_fn_integrity_table(value); + if (table) { + answer = scr_dyn_is_frozen(table); + scr_dyn_release(table); + return answer; + } if (value->kind == SCR_DYN_ARR) return scr_dyn_is_sealed(value) && (value->v.arr.len == 0 || value->v.arr.frozen) && (!value->v.arr.properties || scr_dyn_is_frozen(value->v.arr.properties)); diff --git a/tests/corpus/function-integrity.js b/tests/corpus/function-integrity.js new file mode 100644 index 000000000..4258a7838 --- /dev/null +++ b/tests/corpus/function-integrity.js @@ -0,0 +1,44 @@ +// Object.freeze, seal and preventExtensions on functions, and the queries: +// they act on the function's own properties (name, length, prototype, and +// the ones a program assigns), as in Node. A frozen function still calls, +// and its properties still read. +const createHasher = (block) => { + const hash = (msg) => msg.length * block; + hash.outputLen = 32; + hash.blockLen = block; + hash.create = () => hash(" ".repeat(block)); + return Object.freeze(hash); +}; +const sha = createHasher(64); +console.log(sha("abc"), sha.outputLen, sha.blockLen, sha.create(), sha.name, sha.length); +console.log(Object.isFrozen(sha), Object.isSealed(sha), Object.isExtensible(sha)); +try { + sha.extra = 1; +} catch (error) { + console.log(error.name, error.message); +} +try { + sha.outputLen = 64; +} catch (error) { + console.log(error.name); +} +console.log(sha.outputLen, sha.extra); + +function plain(a, b) { + return a + b; +} +console.log(Object.isFrozen(plain), Object.isSealed(plain), Object.isExtensible(plain)); +plain.tag = "t"; +Object.seal(plain); +console.log(Object.isFrozen(plain), Object.isSealed(plain), Object.isExtensible(plain)); +plain.tag = "u"; +console.log(plain.tag, plain(1, 2)); + +const stopped = () => 0; +Object.preventExtensions(stopped); +console.log(Object.isFrozen(stopped), Object.isSealed(stopped), Object.isExtensible(stopped)); +try { + stopped.late = 1; +} catch (error) { + console.log(error.name, error.message); +} From e50454caf3f8831b1ca47b81566894f894e0c8cd Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 08:48:03 -0500 Subject: [PATCH 09/51] Support TypedArray.of TypedArray.of(...items) on the numeric typed-array constructors creates a new array of its arguments. Each argument converts with ToNumber and then to the element type, in argument order after all arguments are evaluated, as in Node. Arguments that are all numbers copy directly; other values and spreads collect into an array that goes through TypedArray.from's conversion. Spreading a dynamic iterable into a call's items (Array.of, push, TypedArray.of) now drains it through the iterator protocol with the spread call's TypeErrors, and a typed iterable spread into dynamic items boxes and drains the same way. Before, both reached a run-time SC2020. Fixes #1440 Co-authored-by: saboursepideh <25549001+saboursepideh@users.noreply.github.com> --- .../lowering/containers/array-construction.ts | 28 ++++++++ .../src/frontend/lowering/containers/bytes.ts | 64 ++++++++++++++++++- .../tests/corpus/6903-typedarray-of.ts.json | 6 ++ .../corpus/6904-typedarray-of-loose.mjs.json | 6 ++ tests/corpus/6903-typedarray-of.ts | 34 ++++++++++ tests/corpus/6904-typedarray-of-loose.mjs | 37 +++++++++++ 6 files changed, 172 insertions(+), 3 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6903-typedarray-of.ts.json create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6904-typedarray-of-loose.mjs.json create mode 100644 tests/corpus/6903-typedarray-of.ts create mode 100644 tests/corpus/6904-typedarray-of-loose.mjs diff --git a/packages/compiler/src/frontend/lowering/containers/array-construction.ts b/packages/compiler/src/frontend/lowering/containers/array-construction.ts index 83a184400..40e4d6436 100644 --- a/packages/compiler/src/frontend/lowering/containers/array-construction.ts +++ b/packages/compiler/src/frontend/lowering/containers/array-construction.ts @@ -1,6 +1,7 @@ import { generatorDrain } from "../iterator-adapters.js"; import { identityPreservingWidening } from "../coercions/identity.js"; import { liftedFreshSpread } from "../coercions/fresh-spread.js"; +import { checkedIterableSpread } from "../checked-iterable-spread.js"; import { InternalCompilerError } from "../../../errors.js"; import { dynUndefinedExpr, @@ -25,6 +26,7 @@ import { STRING, type SrcLoc, arrayOf, + canConvertToDyn, funcOf, isRefCounted, isSupportedArrayElem, @@ -153,6 +155,32 @@ export function lowerArraySpreadItems( // Fresh elements (an array literal) convert into the element layout. source = liftedFreshSpread(lowerer, node.expression, source, arrType, loc) ?? source; } + // A checked-dynamic source, or any other iterable spread into + // checked-dynamic items, drains once through the iterator protocol + // (with the spread call's TypeErrors), and its elements convert to + // the element type. + const drains = + source.type.kind === "dyn" + ? elem.kind === "dyn" || + elem.kind === "f64" || + elem.kind === "string" || + elem.kind === "bool" + : elem.kind === "dyn" && + !typeEquals(source.type, arrType) && + !( + source.type.kind === "array" && + identityPreservingWidening(lowerer, source.type.elem, elem) + ) && + canConvertToDyn( + source.type, + (id) => lowerer.shapes.get(id), + (id) => lowerer.unions.get(id), + ); + if (drains) { + const spelling = ts.getTextOfNode(node.expression); + const items = checkedIterableSpread(lowerer, source, spelling, loc); + source = lowerer.coerceInto(node.expression, items, arrType); + } if (!typeEquals(source.type, arrType)) { if ( source.type.kind !== "array" || diff --git a/packages/compiler/src/frontend/lowering/containers/bytes.ts b/packages/compiler/src/frontend/lowering/containers/bytes.ts index 8d1d17821..a45d823f1 100644 --- a/packages/compiler/src/frontend/lowering/containers/bytes.ts +++ b/packages/compiler/src/frontend/lowering/containers/bytes.ts @@ -38,7 +38,7 @@ import { lowerOptionalArgument, lowerStaticallyUndefinedArgument, } from "../optional-arguments.js"; -import { lowerCheckedArrayFrom } from "./array-construction.js"; +import { lowerArraySpreadItems, lowerCheckedArrayFrom } from "./array-construction.js"; import { lowerToNumberArgument } from "../lower-exprs.js"; import { lowerBytesIteratorValue } from "../iterator-adapters.js"; import { lowerArrayCallback } from "./callback-arguments.js"; @@ -642,7 +642,7 @@ export function lowerBytesNew( ); } -/** TypedArray.from without a mapping callback. Preserve its iterable / +/** TypedArray.from and TypedArray.of. Preserve from's iterable / * array-like semantics separately from constructor length coercion. */ export function lowerBytesStaticCall( lowerer: Lowerer, @@ -662,10 +662,15 @@ export function lowerBytesStaticCall( loc: locOf(call), }; } - if (access.name.text !== "from" || !ts.isIdentifier(access.expression)) return null; + if ( + (access.name.text !== "from" && access.name.text !== "of") || + !ts.isIdentifier(access.expression) + ) + return null; const symbol = lowerer.resolveValueSymbol(access.expression); const elem = symbol ? own(BYTES_CTORS, ts.symbolName(symbol)) : undefined; if (!elem || !lowerer.isStdlibSymbol(symbol ?? undefined)) return null; + if (access.name.text === "of") return lowerBytesOf(lowerer, call, elem); if ( call.arguments.nodes.length < 1 || call.arguments.nodes.length > 3 || @@ -754,6 +759,59 @@ export function lowerBytesStaticCall( }; } +/** TypedArray.of: a new array of the arguments, each converted with ToNumber + * in argument order after all of them are evaluated. Numbers copy directly; + * other values (and spread items) collect into an array that goes through + * TypedArray.from's conversion, which calls valueOf/toString the same way. */ +function lowerBytesOf(lowerer: Lowerer, call: ts.CallExpression, elem: IrBytesElem): IrExpr { + const loc = locOf(call); + const nodes = call.arguments.nodes; + const type = bytesOf(elem); + if (nodes.some(ts.isSpreadElement)) { + const numeric = nodes.every((node) => { + const mapped = lowerer.mapTypeOf( + lowerer.typeOf(ts.isSpreadElement(node) ? node.expression : node), + ); + return ts.isSpreadElement(node) + ? mapped?.kind === "array" && mapped.elem.kind === "f64" + : mapped?.kind === "f64"; + }); + const elemType = numeric ? F64 : DYN; + const items = lowerArraySpreadItems( + lowerer, + nodes, + elemType, + { kind: "array", elem: elemType }, + loc, + ); + if (numeric) return { kind: "bytesNew", source: items, type, loc }; + return { + kind: "bytesNew", + source: lowerer.coerceToExpected(items, DYN), + from: true, + type, + loc, + }; + } + const values = nodes.map((node) => lowerer.lowerExpr(node)); + if (values.every((value) => value.type.kind === "f64")) { + return { + kind: "bytesNew", + source: { kind: "arrayLit", elems: values, type: arrayOf(F64), loc }, + type, + loc, + }; + } + const elems = values.map((value, i) => lowerer.coerceInto(nodes[i]!, value, DYN)); + return { + kind: "bytesNew", + source: lowerer.coerceToExpected({ kind: "arrayLit", elems, type: arrayOf(DYN), loc }, DYN), + from: true, + type, + loc, + }; +} + /** Construct a view over the complete branded backing allocation. */ function lowerArrayBufferView( lowerer: Lowerer, diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6903-typedarray-of.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6903-typedarray-of.ts.json new file mode 100644 index 000000000..35f4badfb --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6903-typedarray-of.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6903-typedarray-of.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6904-typedarray-of-loose.mjs.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6904-typedarray-of-loose.mjs.json new file mode 100644 index 000000000..ccccc5293 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6904-typedarray-of-loose.mjs.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6904-typedarray-of-loose.mjs" + ], + "diags": [] +} diff --git a/tests/corpus/6903-typedarray-of.ts b/tests/corpus/6903-typedarray-of.ts new file mode 100644 index 000000000..bda6b529a --- /dev/null +++ b/tests/corpus/6903-typedarray-of.ts @@ -0,0 +1,34 @@ +// TypedArray.of: a new array of the arguments, each converted with ToNumber +// and then to the element type, in argument order after all arguments are +// evaluated. Covers each numeric flavor and spread arguments; 6904 covers +// values that are not numbers. +const u8 = Uint8Array.of(1, 2, 3); +console.log(u8.length, u8.byteLength, Array.from(u8).join(",")); +console.log(u8 instanceof Uint8Array); + +console.log(Array.from(Uint8Array.of(-1, 256, 257, 3.7, 0 / 0)).join(",")); +console.log(Array.from(Uint8ClampedArray.of(-1, 256, 1.5, 2.5)).join(",")); +console.log(Array.from(Int8Array.of(127, 128, -129)).join(",")); +console.log(Array.from(Uint16Array.of(65536, -1)).join(",")); +console.log(Array.from(Int16Array.of(32768, -32769)).join(",")); +console.log(Array.from(Uint32Array.of(-1, 4294967296)).join(",")); +console.log(Array.from(Int32Array.of(2147483648, 1.9)).join(",")); +console.log(Array.from(Float32Array.of(0.1, 1e40)).join(",")); +console.log(Array.from(Float64Array.of(0.1, -0)).join(",")); + +const empty = Float64Array.of(); +console.log("empty", empty.length); + +const parts = [4, 5, 6]; +console.log(Array.from(Uint8Array.of(...parts)).join(",")); +console.log(Array.from(Int16Array.of(1, ...parts, 7)).join(",")); + +let order = ""; +function next(n: number): number { + order += n; + return n; +} +console.log(Array.from(Uint8Array.of(next(1), next(2), next(3))).join(","), order); + +const bytes = Uint8Array.of(0xde, 0xad, 0xbe, 0xef); +console.log(Buffer.from(bytes).toString("hex")); diff --git a/tests/corpus/6904-typedarray-of-loose.mjs b/tests/corpus/6904-typedarray-of-loose.mjs new file mode 100644 index 000000000..a067b31c7 --- /dev/null +++ b/tests/corpus/6904-typedarray-of-loose.mjs @@ -0,0 +1,37 @@ +// TypedArray.of over values that are not numbers: each argument goes through +// ToNumber (valueOf / toString) in argument order, after all arguments are +// evaluated, and a Symbol throws a TypeError. +console.log(Array.from(Float64Array.of("12", "", " 0x10 ", "abc", true, false, null, undefined)).join(",")); +console.log(Array.from(Float64Array.of([7], [], {}, [1, 2])).join(",")); +const strings = ["12", " 0x10 ", "abc"]; +console.log(Array.from(Float64Array.of(...strings, 1)).join(",")); +console.log(Array.from(Uint8Array.of("300", true, [2])).join(",")); + +let calls = ""; +const a = { valueOf: () => ((calls += "a"), 9) }; +const b = { toString: () => ((calls += "b"), "8") }; +console.log(Array.from(Uint8Array.of(a, b, 7)).join(","), calls); + +let order = ""; +const v = (n) => ({ valueOf: () => ((order += "v" + n), n) }); +const e = (n) => ((order += "e" + n), v(n)); +console.log(Array.from(Int16Array.of(e(1), e(2))).join(","), order); + +try { + Uint8Array.of(1, Symbol("s")); +} catch (err) { + console.log(err instanceof TypeError); +} + +const dyn = JSON.parse("[1, 2, 300]"); +console.log(Array.from(Uint8Array.of(...dyn)).join(",")); +console.log(Array.from(Uint8Array.of(...dyn, "7", ...[8, 9], ..."12")).join(",")); + +for (const text of ["5", "null", "{}"]) { + const value = JSON.parse(text); + try { + Uint8Array.of(...value); + } catch (err) { + console.log(err instanceof TypeError, err.message); + } +} From f2fd1c85a2c0895d5347c2e14a27a08886be8a92 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 08:50:54 -0500 Subject: [PATCH 10/51] Run Buffer methods on Buffers held as checked-dynamic values --- .../src/backend/executable-features.ts | 2 +- .../compiler/src/backend/llvm/expr-dynamic.ts | 8 +- .../src/backend/native-link-info-core.ts | 4 +- .../compiler/src/backend/native/contracts.ts | 4 +- .../compiler/src/backend/native/executable.ts | 4 +- .../compiler/src/backend/runtime-pack-core.ts | 4 +- .../src/executable/executable-cache.ts | 2 +- packages/compiler/src/index.ts | 2 +- packages/compiler/src/ir/runtime-features.ts | 53 ++- .../compiler/src/native/sanitizer.test.ts | 2 +- .../corpus/6930-dyn-buffer-methods.cjs.json | 6 + .../runtime-pack-matrix.mjs | 4 +- packages/runtime/src/scr_bytes.c | 2 +- packages/runtime/src/scr_bytes.h | 4 + packages/runtime/src/scr_bytes_dyn.c | 376 +++++++++++++++++- packages/runtime/src/scr_dyn.h | 11 +- tests/corpus/6930-dyn-buffer-methods.cjs | 145 +++++++ .../6930-dyn-buffer-methods.cjs.json | 1 + 18 files changed, 595 insertions(+), 39 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6930-dyn-buffer-methods.cjs.json create mode 100644 tests/corpus/6930-dyn-buffer-methods.cjs create mode 100644 tests/harness/corpus-costs/6930-dyn-buffer-methods.cjs.json diff --git a/packages/compiler/src/backend/executable-features.ts b/packages/compiler/src/backend/executable-features.ts index eeb5a76e9..f7cae8508 100644 --- a/packages/compiler/src/backend/executable-features.ts +++ b/packages/compiler/src/backend/executable-features.ts @@ -30,7 +30,7 @@ export function executableLinkFeatures(mod: IrModule): NativeLinkFeatures { ...(features.handleProps ? { handleProps: true } : {}), ...(features.dynForAwait ? { dynForAwait: true } : {}), ...(features.eventsOnceSignal ? { eventsOnceSignal: true } : {}), - ...(features.dynBufferNum ? { dynBufferNum: true } : {}), + ...(features.dynBuffer ? { dynBuffer: true } : {}), ...(features.builtinValues ? { builtinValues: true } : {}), events: features.processEvents, emitter: features.emitter, diff --git a/packages/compiler/src/backend/llvm/expr-dynamic.ts b/packages/compiler/src/backend/llvm/expr-dynamic.ts index 83cbe6d54..f9ebdfd59 100644 --- a/packages/compiler/src/backend/llvm/expr-dynamic.ts +++ b/packages/compiler/src/backend/llvm/expr-dynamic.ts @@ -29,7 +29,7 @@ import { LlvmUnsupportedError } from "./unsupported.js"; import type { LlvmEmitterContext, ExprOf, LlValue } from "./expr-context.js"; import { emitUnionWiden } from "./expr-records.js"; import { emitJsIcMethod, jsIcMethods } from "./js-ic.js"; -import { DYN_BUFFER_NUM_METHODS } from "../../ir/runtime-features.js"; +import { DYN_BUFFER_METHODS } from "../../ir/runtime-features.js"; export function emitDynamicExpr( host: LlvmEmitterContext, @@ -234,11 +234,9 @@ export function emitDynamicExpr( }); argsPtr = arr; } - // Fixed-width Buffer method names take scr_bytes_dyn.c's entry, which + // Buffer.prototype method names take their scr_bytes_dyn.c entry, which // runs them on Buffer receivers and dispatches any other receiver. - const invoke = DYN_BUFFER_NUM_METHODS.has(e.method) - ? "scr_dyn_invoke_buffer_num" - : "scr_dyn_invoke_prepared"; + const invoke = DYN_BUFFER_METHODS.get(e.method) ?? "scr_dyn_invoke_prepared"; host.declare(`declare ptr @${invoke}(ptr, ptr, ptr, ptr, ${host.sizeType}, ptr)`); let name = host.cstr(e.calleeName); if (e.calleeNameValue !== undefined) { diff --git a/packages/compiler/src/backend/native-link-info-core.ts b/packages/compiler/src/backend/native-link-info-core.ts index dca02ef84..a476e1f7f 100644 --- a/packages/compiler/src/backend/native-link-info-core.ts +++ b/packages/compiler/src/backend/native-link-info-core.ts @@ -55,9 +55,9 @@ export interface NativeLinkFeatures { /** events.once with options: scr_events_once_signal.c. Absent (not * false) for programs without it. */ eventsOnceSignal?: boolean; - /** Fixed-width Buffer methods called on checked-dynamic values link + /** Buffer.prototype methods called on checked-dynamic values link * scr_bytes_dyn.c; absent (false) for programs without them. */ - dynBufferNum?: boolean; + dynBuffer?: boolean; /** Builtin functions read as values (Buffer.byteLength, Buffer.alloc, * fs.realpathSync) link scr_builtin_values.c; absent (false) otherwise. */ builtinValues?: boolean; diff --git a/packages/compiler/src/backend/native/contracts.ts b/packages/compiler/src/backend/native/contracts.ts index cd395fc0f..a3c0a62fe 100644 --- a/packages/compiler/src/backend/native/contracts.ts +++ b/packages/compiler/src/backend/native/contracts.ts @@ -123,9 +123,9 @@ export interface CcOptions { /** events.once with options (events.onceSignal): compiles * scr_events_once_signal.c. */ eventsOnceSignal?: boolean; - /** Fixed-width Buffer methods called on checked-dynamic values: + /** Buffer.prototype methods called on checked-dynamic values: * compiles scr_bytes_dyn.c. */ - dynBufferNum?: boolean; + dynBuffer?: boolean; /** Builtin functions read as values: compiles scr_builtin_values.c. */ builtinValues?: boolean; /** The program constructs EventTarget/Event/CustomEvent values or diff --git a/packages/compiler/src/backend/native/executable.ts b/packages/compiler/src/backend/native/executable.ts index 212ccfd98..a551ad600 100644 --- a/packages/compiler/src/backend/native/executable.ts +++ b/packages/compiler/src/backend/native/executable.ts @@ -180,8 +180,8 @@ const RUNTIME_UNITS: readonly RuntimeUnit[] = [ unit((u) => u.opts.assert || u.regex || u.opts.symbol, "scr_assert.c"), unit((u) => u.opts.inspect, "scr_inspect.c", "scr_console_native.c"), unit((u) => u.opts.dynInvoke || u.opts.workers || u.nativeFetch, "scr_dyn_invoke.c"), - // Fixed-width Buffer methods called on checked-dynamic values. - unit((u) => u.opts.dynBufferNum === true && u.opts.dynInvoke, "scr_bytes_dyn.c"), + // Buffer.prototype methods called on checked-dynamic values. + unit((u) => u.opts.dynBuffer === true && u.opts.dynInvoke, "scr_bytes_dyn.c"), // Builtin functions read as values in JavaScript dependencies. unit((u) => u.opts.builtinValues === true, "scr_builtin_values.c"), unit((u) => u.opts.dc, "scr_dc.c"), diff --git a/packages/compiler/src/backend/runtime-pack-core.ts b/packages/compiler/src/backend/runtime-pack-core.ts index 00558876b..b59fdc093 100644 --- a/packages/compiler/src/backend/runtime-pack-core.ts +++ b/packages/compiler/src/backend/runtime-pack-core.ts @@ -88,7 +88,7 @@ export interface RuntimeFeatureSet extends NativeLinkFeatures { handleProps: boolean; dynForAwait: boolean; eventsOnceSignal: boolean; - dynBufferNum: boolean; + dynBuffer: boolean; builtinValues: boolean; warningReplace: boolean; stdioExtended: boolean; @@ -285,7 +285,7 @@ export function effectiveRuntimeFeatures( handleProps: features.handleProps === true, dynForAwait: features.dynForAwait === true, eventsOnceSignal: features.eventsOnceSignal === true, - dynBufferNum: features.dynBufferNum === true, + dynBuffer: features.dynBuffer === true, builtinValues: features.builtinValues === true, warningReplace: features.warningReplace === true, stdioExtended: features.stdioExtended === true, diff --git a/packages/compiler/src/executable/executable-cache.ts b/packages/compiler/src/executable/executable-cache.ts index f9d4688ae..806773c4c 100644 --- a/packages/compiler/src/executable/executable-cache.ts +++ b/packages/compiler/src/executable/executable-cache.ts @@ -70,7 +70,7 @@ export interface EarlyExecutableNativeFeatures { handleProps?: boolean; dynForAwait?: boolean; eventsOnceSignal?: boolean; - dynBufferNum?: boolean; + dynBuffer?: boolean; builtinValues?: boolean; events: boolean; emitter: boolean; diff --git a/packages/compiler/src/index.ts b/packages/compiler/src/index.ts index e9b8514af..02aac4262 100644 --- a/packages/compiler/src/index.ts +++ b/packages/compiler/src/index.ts @@ -558,7 +558,7 @@ async function compileExecutableNative( ...(features.handleProps ? { handleProps: true } : {}), ...(features.dynForAwait ? { dynForAwait: true } : {}), ...(features.eventsOnceSignal ? { eventsOnceSignal: true } : {}), - ...(features.dynBufferNum ? { dynBufferNum: true } : {}), + ...(features.dynBuffer ? { dynBuffer: true } : {}), ...(features.builtinValues ? { builtinValues: true } : {}), events: features.events, emitter: features.emitter, diff --git a/packages/compiler/src/ir/runtime-features.ts b/packages/compiler/src/ir/runtime-features.ts index 9c6b08a96..4142b6efe 100644 --- a/packages/compiler/src/ir/runtime-features.ts +++ b/packages/compiler/src/ir/runtime-features.ts @@ -10,9 +10,7 @@ import { import { everyModuleNode } from "./traverse.js"; import { JsIcSites, hasJsIcSite } from "./js-ic.js"; -/** The fixed-width Buffer read/write method names (both "UInt" and "Uint" - * spellings) a checked-dynamic method call routes to scr_bytes_dyn.c. */ -export const DYN_BUFFER_NUM_METHODS: ReadonlySet = new Set( +const DYN_BUFFER_NUM_METHODS: ReadonlySet = new Set( ["read", "write"].flatMap((rw) => [ ...["UInt", "Uint"].flatMap((u) => ["8", "16LE", "16BE", "32LE", "32BE"].map((w) => `${rw}${u}${w}`), @@ -31,6 +29,45 @@ export const DYN_BUFFER_NUM_METHODS: ReadonlySet = new Set( ]), ); +/** Buffer.prototype members scr_bytes_dyn.c refuses on Buffer receivers. */ +const DYN_BUFFER_REFUSED_METHODS: readonly string[] = [ + ...["read", "write"].flatMap((rw) => [ + ...["UIntLE", "UIntBE", "UintLE", "UintBE", "IntLE", "IntBE"].map((n) => `${rw}${n}`), + ...["BigUInt64", "BigUint64", "BigInt64"].flatMap((n) => [`${rw}${n}LE`, `${rw}${n}BE`]), + ]), + ...["ascii", "base64", "base64url", "latin1", "hex", "ucs2", "utf8"].flatMap((enc) => [ + `${enc}Slice`, + `${enc}Write`, + ]), + "inspect", +]; + +/** The scr_bytes_dyn.c entry, one per method family, that a checked-dynamic + * call of a Buffer.prototype method name routes to: the fixed-width reads and + * writes (both "UInt" and "Uint" spellings), write, fill, toString, the + * searches, and the other Buffer-only members. Each runs the method on a + * Buffer receiver and gives any other receiver the ordinary dispatch. */ +export const DYN_BUFFER_METHODS: ReadonlyMap = new Map([ + ...[...DYN_BUFFER_NUM_METHODS].map((m): [string, string] => [m, "scr_dyn_invoke_buffer_num"]), + ["write", "scr_dyn_invoke_buffer_write"], + ["fill", "scr_dyn_invoke_buffer_fill"], + ["toString", "scr_dyn_invoke_buffer_to_string"], + ...["indexOf", "lastIndexOf", "includes"].map((m): [string, string] => [ + m, + "scr_dyn_invoke_buffer_search", + ]), + ...[ + "toJSON", + "equals", + "compare", + "copy", + "swap16", + "swap32", + "swap64", + ...DYN_BUFFER_REFUSED_METHODS, + ].map((m): [string, string] => [m, "scr_dyn_invoke_buffer"]), +]); + const DYN_ASYNC_LIB_FNS: ReadonlySet = new Set([ // DEP0040 rides the dyn-async unit's warning machinery. "async.awaitDyn", @@ -694,10 +731,10 @@ const RUNTIME_FEATURE_RULES = { handleProps: rule({ fns: ["child.propsInstall"] }), /** for await over checked-dynamic values (scr_dyn_for_await.c). */ dynForAwait: rule({ fns: ["dyn.asyncIteratorOpen", "dyn.asyncIteratorWasSync"] }), - /** Fixed-width Buffer read/write methods called on checked-dynamic - * values (scr_bytes_dyn.c). */ - dynBufferNum: rule({ - expr: (node, _tables) => node.kind === "dynInvoke" && DYN_BUFFER_NUM_METHODS.has(node.method), + /** Buffer.prototype methods called on checked-dynamic values + * (scr_bytes_dyn.c). */ + dynBuffer: rule({ + expr: (node, _tables) => node.kind === "dynInvoke" && DYN_BUFFER_METHODS.has(node.method), }), /** Builtin functions read as values (scr_builtin_values.c). */ builtinValues: rule({ fns: ["buffer.byteLenValue", "buffer.allocValue", "fs.realpathValue"] }), @@ -898,7 +935,7 @@ function noRuntimeFeatures(): RuntimeFeatures { streamDyn: false, handleProps: false, dynForAwait: false, - dynBufferNum: false, + dynBuffer: false, builtinValues: false, symbol: false, tls: false, diff --git a/packages/compiler/src/native/sanitizer.test.ts b/packages/compiler/src/native/sanitizer.test.ts index 8fd2d0858..e6beb24cc 100644 --- a/packages/compiler/src/native/sanitizer.test.ts +++ b/packages/compiler/src/native/sanitizer.test.ts @@ -81,7 +81,7 @@ test.skipIf(process.platform !== "linux" && process.platform !== "darwin")( handleProps: false, dynForAwait: false, eventsOnceSignal: false, - dynBufferNum: false, + dynBuffer: false, builtinValues: false, warningReplace: false, stdioExtended: false, diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6930-dyn-buffer-methods.cjs.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6930-dyn-buffer-methods.cjs.json new file mode 100644 index 000000000..7334e8458 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6930-dyn-buffer-methods.cjs.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6930-dyn-buffer-methods.cjs" + ], + "diags": [] +} diff --git a/packages/runtime-pack-common/runtime-pack-matrix.mjs b/packages/runtime-pack-common/runtime-pack-matrix.mjs index 41ed81579..0f40218a1 100644 --- a/packages/runtime-pack-common/runtime-pack-matrix.mjs +++ b/packages/runtime-pack-common/runtime-pack-matrix.mjs @@ -61,8 +61,8 @@ export const OPTIONAL_RUNTIME_SOURCES = [ ["scr_inspect.c", "inspect"], ["scr_console_native.c", "inspect"], ["scr_dyn_invoke.c", any("dynInvoke", "nativeFetch")], - // Fixed-width Buffer methods called on checked-dynamic values. - ["scr_bytes_dyn.c", all("dynBufferNum", "dynInvoke")], + // Buffer.prototype methods called on checked-dynamic values. + ["scr_bytes_dyn.c", all("dynBuffer", "dynInvoke")], // Builtin functions read as values in JavaScript dependencies. ["scr_builtin_values.c", "builtinValues"], ["scr_dc.c", "dc"], diff --git a/packages/runtime/src/scr_bytes.c b/packages/runtime/src/scr_bytes.c index 2e964e78b..6262b2530 100644 --- a/packages/runtime/src/scr_bytes.c +++ b/packages/runtime/src/scr_bytes.c @@ -2519,7 +2519,7 @@ static bool scr_enc_eq_ci(const ScrStr *raw, const char *name) { return true; } -static ScrStr *scr_bytes_normalize_encoding(const ScrStr *raw) { +ScrStr *scr_bytes_normalize_encoding(const ScrStr *raw) { static const struct { const char *from; const char *to; } map[] = { {"utf8", "utf8"}, {"utf-8", "utf8"}, {"hex", "hex"}, {"base64", "base64"}, {"base64url", "base64url"}, diff --git a/packages/runtime/src/scr_bytes.h b/packages/runtime/src/scr_bytes.h index 8080751ad..9c81caea5 100644 --- a/packages/runtime/src/scr_bytes.h +++ b/packages/runtime/src/scr_bytes.h @@ -150,6 +150,10 @@ ScrStr *scr_strdec_write(const ScrStr *enc, double pending, const ScrBytes *chun double scr_strdec_next(const ScrStr *enc, double pending, const ScrBytes *chunk); ScrStr *scr_strdec_end(const ScrStr *enc, double pending); ScrStr *scr_strdec_encoding(const ScrDyn *enc); /* +1, or NULL with ERR_UNKNOWN_ENCODING pending */ +/* A Buffer encoding name known only at run time, Node's getEncodingOps: the + * canonical name of an encoding in any ASCII case (+1), or NULL with + * ERR_UNKNOWN_ENCODING ("Unknown encoding: ") pending. Borrows raw. */ +ScrStr *scr_bytes_normalize_encoding(const ScrStr *raw); /* `new Uint8Array(n)` / Buffer.alloc(n): zero-filled. n goes through * ToIndex like JS (NaN → 0, truncate toward zero; negative or > 2^53-1 diff --git a/packages/runtime/src/scr_bytes_dyn.c b/packages/runtime/src/scr_bytes_dyn.c index d4b29574b..58130a072 100644 --- a/packages/runtime/src/scr_bytes_dyn.c +++ b/packages/runtime/src/scr_bytes_dyn.c @@ -1,8 +1,9 @@ -/* The fixed-width read/write methods of a Buffer held as a checked-dynamic - * value: a JavaScript dependency's `buf.writeUInt32LE(v, 0)` on a buffer it - * got back untyped (yazl). The compiler routes only the dyn method calls - * that name one of these methods here, so programs without them link - * nothing new; any other receiver takes the ordinary dispatch. */ +/* The Buffer.prototype methods of a Buffer held as a checked-dynamic value: + * a JavaScript dependency's `buf.writeUInt32LE(v, 0)` on a buffer it got back + * untyped (yazl), or `this.buf.write(s)` on a buffer a constructor function + * stored. The compiler routes only the dyn method calls that name one of + * these methods here, so programs without them link nothing new; any other + * receiver takes the ordinary dispatch. */ #include "scr_runtime.h" #include @@ -87,14 +88,373 @@ static ScrDyn *scr_bytes_dyn_num_method(ScrBytes *b, bool read, size_t at, ScrDy return scr_exc_pending() ? NULL : scr_dyn_new_num(out); } +static const ScrDyn *scr_bytes_dyn_arg(ScrDyn *const *args, size_t argc, size_t i) { + return i < argc ? args[i] : scr_dyn_undefined(); +} + +static bool scr_bytes_dyn_undef(const ScrDyn *d) { return d->kind == SCR_DYN_UNDEF; } + +static ScrDyn *scr_bytes_dyn_str(const char *text, size_t len) { + ScrStr *s = scr_str_new(text, len); + ScrDyn *out = scr_dyn_new_str(s); + scr_str_release(s); + return out; +} + +/* Node's validateOffset: a number (ERR_INVALID_ARG_TYPE otherwise), then an + * integer in [0, max] (ERR_OUT_OF_RANGE). */ +static bool scr_bytes_dyn_offset(const ScrDyn *v, const char *name, double max, double *out) { + if (v->kind != SCR_DYN_NUM) { + scr_dyn_arg_type_fail(name, "of type number", v); + return false; + } + if (!scr_bytes_validate_off(name, v->v.num, max)) return false; + *out = v->v.num; + return true; +} + +/* getEncodingOps over an argument: ToString (`encoding += ''`), then the + * canonical name in any ASCII case. +1, or NULL with the exception pending. */ +static ScrStr *scr_bytes_dyn_encoding(const ScrDyn *v) { + ScrStr *raw = scr_dyn_string_coerce_js(v); + if (!raw) return NULL; + ScrStr *out = scr_bytes_normalize_encoding(raw); + scr_str_release(raw); + return out; +} + +/* Node's toInteger(n, 0) for copy's offsets: an integer stays, other finite + * numbers in the safe range floor, everything else (NaN, ±Infinity) is 0. */ +static bool scr_bytes_dyn_to_integer(const ScrDyn *v, double *out) { + double n; + if (!scr_dyn_number_coerce_js(v, &n)) return false; + *out = n == n && n >= -9007199254740991.0 && n <= 9007199254740991.0 ? floor(n) : 0; + return true; +} + +/* buf.write(string[, offset[, length]][, encoding]): buffer.js's overload + * resolution and offset checks, then the encoding, then the string type. */ +static ScrDyn *scr_bytes_dyn_write(ScrBytes *b, ScrDyn *const *args, size_t argc) { + const ScrDyn *string = scr_bytes_dyn_arg(args, argc, 0); + const ScrDyn *offset = scr_bytes_dyn_arg(args, argc, 1); + const ScrDyn *length = scr_bytes_dyn_arg(args, argc, 2); + const ScrDyn *encoding = scr_bytes_dyn_arg(args, argc, 3); + double at = 0, budget = (double)b->len; + ScrStr *enc = NULL; + if (scr_bytes_dyn_undef(offset)) { + encoding = scr_dyn_undefined(); + } else if (scr_bytes_dyn_undef(length) && offset->kind == SCR_DYN_STR) { + encoding = offset; + } else { + if (!scr_bytes_dyn_offset(offset, "offset", (double)b->len, &at)) return NULL; + budget = (double)b->len - at; + if (length->kind == SCR_DYN_STR) { + encoding = length; + } else if (!scr_bytes_dyn_undef(length)) { + double n; + if (!scr_bytes_dyn_offset(length, "length", (double)b->len, &n)) return NULL; + if (n < budget) budget = n; + } + } + if (!scr_dyn_truthy(encoding)) { + enc = scr_str_new("utf8", 4); + } else { + enc = scr_bytes_dyn_encoding(encoding); + if (!enc) return NULL; + } + if (string->kind != SCR_DYN_STR) { + scr_str_release(enc); + static const char message[] = "argument must be a string"; + scr_throw_error_msg_code(SCR_ERR_TYPE, message, sizeof message - 1, "ERR_INVALID_ARG_TYPE"); + return NULL; + } + double written = scr_bytes_write_str(b, string->v.str, enc, at, budget, true); + scr_str_release(enc); + return scr_exc_pending() ? NULL : scr_dyn_new_num(written); +} + +/* Node's inspect quoting for the ERR_INVALID_ARG_VALUE render of a string: + * single quotes unless the text has one, then double, then backticks. */ +static ScrStr *scr_bytes_dyn_quoted(const ScrStr *text) { + char quote = '\''; + if (memchr(text->data, '\'', text->len)) quote = memchr(text->data, '"', text->len) ? '`' : '"'; + ScrJsonBuf jb; + scr_jb_init(&jb); + scr_jb_puts(&jb, "The argument 'value' is invalid. Received "); + scr_jb_putc(&jb, quote); + scr_jb_put_str(&jb, text); + scr_jb_putc(&jb, quote); + return scr_jb_finish(&jb); +} + +/* buf.fill(value[, offset[, end]][, encoding]): buffer.js's _fill. A string + * takes its encoding from the first string among offset and end; numbers and + * other non-view values fill with ToUint32(value) & 255. Answers the receiver. */ +static ScrDyn *scr_bytes_dyn_fill(ScrDyn *recv, ScrDyn *const *args, size_t argc) { + ScrBytes *b = recv->v.bytes; + const ScrDyn *value = scr_bytes_dyn_arg(args, argc, 0); + const ScrDyn *offset = scr_bytes_dyn_arg(args, argc, 1); + const ScrDyn *end = scr_bytes_dyn_arg(args, argc, 2); + ScrStr *enc = NULL; + bool whole = false; + if (value->kind == SCR_DYN_STR) { + const ScrDyn *encoding = scr_bytes_dyn_arg(args, argc, 3); + if (scr_bytes_dyn_undef(offset) || offset->kind == SCR_DYN_STR) { + encoding = offset; + whole = true; + } else if (end->kind == SCR_DYN_STR) { + encoding = end; + end = scr_dyn_undefined(); + } + if (encoding->kind == SCR_DYN_UNDEF || encoding->kind == SCR_DYN_NULL || + (encoding->kind == SCR_DYN_STR && encoding->v.str->len == 0)) { + enc = scr_str_new("utf8", 4); + } else if (encoding->kind != SCR_DYN_STR) { + scr_dyn_arg_type_fail("encoding", "of type string", encoding); + return NULL; + } else { + enc = scr_bytes_normalize_encoding(encoding->v.str); + if (!enc) return NULL; + } + } + double from = 0, to = (double)b->len; + if (!whole && !scr_bytes_dyn_undef(offset)) { + if (!scr_bytes_dyn_offset(offset, "offset", 9007199254740991.0, &from) || + (!scr_bytes_dyn_undef(end) && !scr_bytes_dyn_offset(end, "end", (double)b->len, &to))) { + scr_str_release(enc); + return NULL; + } + if (from >= to) { + scr_str_release(enc); + return scr_dyn_retain(recv); + } + } + ScrBytes *out; + if (enc) { + const ScrStr *text = value->v.str; + ScrBytes *pattern = scr_bytes_from_str(text, enc); + scr_str_release(enc); + if (text->len > 0 && pattern->len == 0) { + scr_bytes_release(pattern); + ScrStr *message = scr_bytes_dyn_quoted(text); + scr_throw_error_msg_code(SCR_ERR_TYPE, message->data, message->len, "ERR_INVALID_ARG_VALUE"); + scr_str_release(message); + return NULL; + } + /* An empty string fills with zeros. */ + out = text->len == 0 ? scr_bytes_fill_num(b, 0, 2, from, to) : scr_bytes_fill(b, pattern, 2, from, to); + scr_bytes_release(pattern); + } else if (value->kind == SCR_DYN_BYTES && !value->v.bytes->is_data_view && + value->v.bytes->elem == SCR_BYTES_U8) { + out = scr_bytes_fill(b, value->v.bytes, 2, from, to); + } else { + double n; + if (!scr_dyn_number_coerce_js(value, &n)) return NULL; + out = scr_bytes_fill_num(b, n, 2, from, to); + } + if (!out) return NULL; + scr_bytes_release(out); + return scr_dyn_retain(recv); +} +/* buf.toString(encoding, start, end): the window clamps to + * [0, length] with NaN as 0, an empty window answers "" before the + * encoding resolves, and an undefined encoding is utf8. */ +static ScrDyn *scr_bytes_dyn_to_string(ScrBytes *b, ScrDyn *const *args, size_t argc) { + const ScrDyn *encoding = scr_bytes_dyn_arg(args, argc, 0); + const ScrDyn *start = scr_bytes_dyn_arg(args, argc, 1); + const ScrDyn *end = scr_bytes_dyn_arg(args, argc, 2); + double from = 0, to = (double)b->len; + if (!scr_bytes_dyn_undef(start) && !scr_dyn_number_coerce_js(start, &from)) return NULL; + if (!scr_bytes_dyn_undef(end) && !scr_dyn_number_coerce_js(end, &to)) return NULL; + if (from != from) from = 0; + if (to != to) to = 0; + from = trunc(from); + to = trunc(to); + if (from >= (double)b->len || to <= from || to <= 0) return scr_bytes_dyn_str("", 0); + ScrStr *enc = scr_bytes_dyn_undef(encoding) ? scr_str_new("utf8", 4) : scr_bytes_dyn_encoding(encoding); + if (!enc) return NULL; + ScrStr *text = scr_bytes_to_str_range(b, enc, from, to); + scr_str_release(enc); + ScrDyn *out = scr_dyn_new_str(text); + scr_str_release(text); + return out; +} + +/* buf.toJSON(): { type: 'Buffer', data: [...bytes] }. */ +static ScrDyn *scr_bytes_dyn_to_json(const ScrBytes *b) { + ScrDyn *data = scr_dyn_new_arr(); + for (size_t i = 0; i < b->len; i++) scr_dyn_arr_push(data, scr_dyn_new_num(scr_bytes_get(b, (double)i))); + ScrDyn *out = scr_dyn_new_obj(); + scr_dyn_obj_set(out, "type", 4, scr_bytes_dyn_str("Buffer", 6)); + scr_dyn_obj_set(out, "data", 4, data); + return out; +} + +/* buf.copy(target[, targetStart[, sourceStart[, sourceEnd]]]): buffer.js's + * _copy offsets (toInteger, then the >= 0 and source-length checks) into a + * Buffer or Uint8Array. Other views have no lowering here. */ +static ScrDyn *scr_bytes_dyn_copy(const ScrBytes *b, ScrDyn *const *args, size_t argc) { + const ScrDyn *target = scr_bytes_dyn_arg(args, argc, 0); + if (target->kind != SCR_DYN_BYTES) { + scr_dyn_arg_type_fail("target", "an instance of Buffer or Uint8Array", target); + return NULL; + } + if (target->v.bytes->is_data_view || target->v.bytes->elem != SCR_BYTES_U8) { + static const char message[] = + "'Buffer.prototype.copy' into a view that is not a Buffer or Uint8Array on a dynamic value is not supported yet"; + scr_throw_error_msg(SCR_ERR_ERROR, message, sizeof message - 1); + return NULL; + } + double offsets[3] = { 0, 0, (double)b->len }; + for (size_t i = 0; i < 3; i++) { + const ScrDyn *v = scr_bytes_dyn_arg(args, argc, i + 1); + if (!scr_bytes_dyn_undef(v) && !scr_bytes_dyn_to_integer(v, &offsets[i])) return NULL; + } + return scr_dyn_new_num(scr_bytes_copy_into(b, target->v.bytes, 3, offsets[0], offsets[1], offsets[2])); +} + +/* buf.indexOf/lastIndexOf/includes(value[, byteOffset][, encoding]): + * buffer.js's bidirectionalIndexOf. A string byteOffset is the encoding; + * the offset clamps to the int32 range, and NaN searches everything. */ +static ScrDyn *scr_bytes_dyn_index_of(const ScrBytes *b, ScrDyn *const *args, size_t argc, bool fwd, + bool includes) { + const ScrDyn *value = scr_bytes_dyn_arg(args, argc, 0); + const ScrDyn *offset = scr_bytes_dyn_arg(args, argc, 1); + const ScrDyn *encoding = scr_bytes_dyn_arg(args, argc, 2); + double at = NAN; + if (offset->kind == SCR_DYN_STR) { + encoding = offset; + } else if (!scr_bytes_dyn_undef(offset)) { + if (!scr_dyn_number_coerce_js(offset, &at)) return NULL; + if (at > 2147483647.0) at = 2147483647.0; + else if (at < -2147483648.0) at = -2147483648.0; + } + double found; + if (value->kind == SCR_DYN_NUM) { + found = scr_bytes_index_of_num(b, value->v.num, at, fwd); + } else { + ScrStr *enc = NULL; + if (!scr_bytes_dyn_undef(encoding)) { + ScrStr *raw = scr_dyn_string_coerce_js(encoding); + if (!raw) return NULL; + /* An unknown encoding throws for a string needle only. */ + enc = value->kind == SCR_DYN_STR ? scr_bytes_normalize_encoding(raw) : NULL; + scr_str_release(raw); + if (value->kind == SCR_DYN_STR && !enc) return NULL; + } + if (value->kind == SCR_DYN_STR) { + if (!enc) enc = scr_str_new("utf8", 4); + ScrBytes *needle = scr_bytes_from_str(value->v.str, enc); + found = scr_bytes_index_of(b, needle, at, enc->len == 7 && !memcmp(enc->data, "utf16le", 7) ? 2 : 1, fwd); + scr_bytes_release(needle); + scr_str_release(enc); + } else if (value->kind == SCR_DYN_BYTES && !value->v.bytes->is_data_view && + value->v.bytes->elem == SCR_BYTES_U8) { + found = scr_bytes_index_of(b, value->v.bytes, at, 1, fwd); + } else { + scr_dyn_arg_type_fail("value", "one of type number or string or an instance of Buffer or Uint8Array", value); + return NULL; + } + } + return includes ? scr_dyn_new_bool(found != -1) : scr_dyn_new_num(found); +} + +/* Buffer.prototype members this dispatch does not run: an explicit refusal + * instead of the "is not a function" a missing method would give. */ +static bool scr_bytes_dyn_refused(const char *method) { + static const char *const prefixes[] = { "readBig", "writeBig", "readUInt", "readUint", "readInt", "writeUInt", + "writeUint", "writeInt", "readFloat", "readDouble", "writeFloat", + "writeDouble" }; + static const char *const names[] = { "asciiSlice", "base64Slice", "base64urlSlice", "latin1Slice", "hexSlice", + "ucs2Slice", "utf8Slice", "asciiWrite", "base64Write", "base64urlWrite", + "latin1Write", "hexWrite", "ucs2Write", "utf8Write", "inspect" }; + for (size_t i = 0; i < sizeof prefixes / sizeof prefixes[0]; i++) + if (!strncmp(method, prefixes[i], strlen(prefixes[i]))) return true; + for (size_t i = 0; i < sizeof names / sizeof names[0]; i++) + if (!strcmp(method, names[i])) return true; + return false; +} + +static bool scr_bytes_dyn_buffer(const ScrDyn *recv) { + return recv->kind == SCR_DYN_BYTES && recv->buffer && !recv->v.bytes->is_data_view; +} + +/* One entry per method family, so a program links only the families it + * names: each runs its methods on a Buffer receiver and gives any other + * receiver the ordinary dispatch. */ ScrDyn *scr_dyn_invoke_buffer_num(ScrDyn *recv, ScrDyn *callee, const char *method, ScrDyn *const *args, size_t argc, const char *what) { - if (recv->kind == SCR_DYN_BYTES && recv->buffer && !recv->v.bytes->is_data_view) { + if (scr_bytes_dyn_buffer(recv)) { bool read = !strncmp(method, "read", 4); const char *rest = method + (read ? 4 : 5); for (size_t at = 0; at < sizeof scr_bytes_dyn_names / sizeof scr_bytes_dyn_names[0]; at++) { - if (!strcmp(rest, scr_bytes_dyn_names[at].name)) - return scr_bytes_dyn_num_method(recv->v.bytes, read, at, args, argc); + if (!strcmp(rest, scr_bytes_dyn_names[at].name)) return scr_bytes_dyn_num_method(recv->v.bytes, read, at, args, argc); + } + } + return scr_dyn_invoke_prepared(recv, callee, method, args, argc, what); +} + +ScrDyn *scr_dyn_invoke_buffer_write(ScrDyn *recv, ScrDyn *callee, const char *method, ScrDyn *const *args, + size_t argc, const char *what) { + if (scr_bytes_dyn_buffer(recv)) return scr_bytes_dyn_write(recv->v.bytes, args, argc); + return scr_dyn_invoke_prepared(recv, callee, method, args, argc, what); +} + +ScrDyn *scr_dyn_invoke_buffer_fill(ScrDyn *recv, ScrDyn *callee, const char *method, ScrDyn *const *args, + size_t argc, const char *what) { + if (scr_bytes_dyn_buffer(recv)) return scr_bytes_dyn_fill(recv, args, argc); + return scr_dyn_invoke_prepared(recv, callee, method, args, argc, what); +} + +ScrDyn *scr_dyn_invoke_buffer_to_string(ScrDyn *recv, ScrDyn *callee, const char *method, ScrDyn *const *args, + size_t argc, const char *what) { + if (scr_bytes_dyn_buffer(recv)) return scr_bytes_dyn_to_string(recv->v.bytes, args, argc); + return scr_dyn_invoke_prepared(recv, callee, method, args, argc, what); +} + +/* indexOf, lastIndexOf and includes. */ +ScrDyn *scr_dyn_invoke_buffer_search(ScrDyn *recv, ScrDyn *callee, const char *method, ScrDyn *const *args, + size_t argc, const char *what) { + if (scr_bytes_dyn_buffer(recv)) { + bool includes = !strcmp(method, "includes"); + return scr_bytes_dyn_index_of(recv->v.bytes, args, argc, strcmp(method, "lastIndexOf") != 0, includes); + } + return scr_dyn_invoke_prepared(recv, callee, method, args, argc, what); +} + +/* toJSON, equals, compare, copy, swap16/32/64, and the refused members. */ +ScrDyn *scr_dyn_invoke_buffer(ScrDyn *recv, ScrDyn *callee, const char *method, ScrDyn *const *args, + size_t argc, const char *what) { + if (scr_bytes_dyn_buffer(recv)) { + ScrBytes *b = recv->v.bytes; + if (!strcmp(method, "toJSON")) return scr_bytes_dyn_to_json(b); + if (!strcmp(method, "equals")) { + bool same = scr_bytes_equals_chk(b, scr_bytes_dyn_arg(args, argc, 0)); + return scr_exc_pending() ? NULL : scr_dyn_new_bool(same); + } + if (!strcmp(method, "compare")) { + double order = scr_bytes_compare_chk(b, scr_bytes_dyn_arg(args, argc, 0), scr_bytes_dyn_arg(args, argc, 1), + scr_bytes_dyn_arg(args, argc, 2), scr_bytes_dyn_arg(args, argc, 3), + scr_bytes_dyn_arg(args, argc, 4)); + return scr_exc_pending() ? NULL : scr_dyn_new_num(order); + } + if (!strcmp(method, "copy")) return scr_bytes_dyn_copy(b, args, argc); + if (!strcmp(method, "swap16") || !strcmp(method, "swap32") || !strcmp(method, "swap64")) { + ScrBytes *swapped = scr_bytes_swap(b, method[4] == '1' ? 2 : method[4] == '3' ? 4 : 8); + if (!swapped) return NULL; + scr_bytes_release(swapped); + return scr_dyn_retain(recv); + } + if (scr_bytes_dyn_refused(method)) { + ScrJsonBuf jb; + scr_jb_init(&jb); + scr_jb_puts(&jb, "'Buffer.prototype."); + scr_jb_puts(&jb, method); + scr_jb_puts(&jb, "' on a dynamic value is not supported yet"); + ScrStr *message = scr_jb_finish(&jb); + scr_throw_error_msg(SCR_ERR_ERROR, message->data, message->len); + scr_str_release(message); + return NULL; } } return scr_dyn_invoke_prepared(recv, callee, method, args, argc, what); diff --git a/packages/runtime/src/scr_dyn.h b/packages/runtime/src/scr_dyn.h index a31f7c49e..45c0b9fbf 100644 --- a/packages/runtime/src/scr_dyn.h +++ b/packages/runtime/src/scr_dyn.h @@ -1153,10 +1153,15 @@ bool scr_dyn_strict_eq(const ScrDyn *a, const ScrDyn *b); * result owned (+1). MAY THROW (NULL with the exception pending). */ ScrDyn *scr_dyn_prepare_method(ScrDyn *recv, const char *method); ScrDyn *scr_dyn_invoke_prepared(ScrDyn *recv, ScrDyn *callee, const char *method, ScrDyn *const *args, size_t argc, const char *what); -/* scr_dyn_invoke_prepared for the fixed-width Buffer read/write method names - * (scr_bytes_dyn.c): a Buffer receiver runs the method, any other takes the - * ordinary dispatch. */ +/* scr_dyn_invoke_prepared for Buffer.prototype method names (scr_bytes_dyn.c), + * one entry per method family: a Buffer receiver runs the method, any other + * takes the ordinary dispatch. */ ScrDyn *scr_dyn_invoke_buffer_num(ScrDyn *recv, ScrDyn *callee, const char *method, ScrDyn *const *args, size_t argc, const char *what); +ScrDyn *scr_dyn_invoke_buffer_write(ScrDyn *recv, ScrDyn *callee, const char *method, ScrDyn *const *args, size_t argc, const char *what); +ScrDyn *scr_dyn_invoke_buffer_fill(ScrDyn *recv, ScrDyn *callee, const char *method, ScrDyn *const *args, size_t argc, const char *what); +ScrDyn *scr_dyn_invoke_buffer_to_string(ScrDyn *recv, ScrDyn *callee, const char *method, ScrDyn *const *args, size_t argc, const char *what); +ScrDyn *scr_dyn_invoke_buffer_search(ScrDyn *recv, ScrDyn *callee, const char *method, ScrDyn *const *args, size_t argc, const char *what); +ScrDyn *scr_dyn_invoke_buffer(ScrDyn *recv, ScrDyn *callee, const char *method, ScrDyn *const *args, size_t argc, const char *what); ScrDyn *scr_dyn_invoke(ScrDyn *recv, const char *method, ScrDyn *const *args, size_t argc, const char *what); /* Keyed read on a FUNC node: the own-property table first (defineProperties * writes land there), then "name" (the box's best-effort static name; "" diff --git a/tests/corpus/6930-dyn-buffer-methods.cjs b/tests/corpus/6930-dyn-buffer-methods.cjs new file mode 100644 index 000000000..d9870d274 --- /dev/null +++ b/tests/corpus/6930-dyn-buffer-methods.cjs @@ -0,0 +1,145 @@ +// Buffer methods on Buffers held as checked-dynamic values: a buffer stored +// on `this` by a constructor function, or assigned later, keeps every +// Buffer.prototype method. The same names on other receivers keep their own. +"use strict"; + +function Writer(size) { + this.buf = Buffer.alloc(size); + this.at = 0; +} +Writer.prototype.put = function (text, encoding) { + this.at += this.buf.write(text, this.at, encoding); + return this; +}; + +const w = new Writer(12); +w.put("ab").put("6364", "hex").put("é", "latin1"); +console.log(w.at, w.buf.toString("hex", 0, w.at), w.buf.toString("latin1", 0, w.at)); + +const t = (name, f) => { + try { + const v = f(); + console.log(name, Buffer.isBuffer(v) ? `<${v.toString("hex")}>` : JSON.stringify(v)); + } catch (e) { + console.log(name, e.name, e.code, e.message); + } +}; + +function Holder() { + this.b = Buffer.from("hello world"); +} +const h = new Holder(); +const fresh = () => { + h.b = Buffer.from("hello world"); + return h.b; +}; + +// write overloads, offset and length checks, encodings. +t("write", () => fresh() && h.b.write("HE")); +t("write offset", () => fresh() && [h.b.write("zz", 3), h.b.toString()]); +t("write offset length", () => fresh() && [h.b.write("abcdef", 1, 2), h.b.toString()]); +t("write offset encoding", () => fresh() && [h.b.write("6869", 5, "hex"), h.b.toString()]); +t("write encoding", () => fresh() && [h.b.write("aGk=", "base64"), h.b.toString()]); +t("write length encoding", () => fresh() && [h.b.write("abc", 0, 2, "latin1"), h.b.toString()]); +t("write undefined offset", () => fresh() && [h.b.write("zz", undefined, 1), h.b.toString()]); +t("write null encoding", () => fresh() && h.b.write("ab", 0, 2, null)); +t("write case", () => fresh() && h.b.write("é", 0, 2, "UTF-8")); +t("write split code point", () => fresh() && h.b.write("\u{1F600}", 9)); +t("write not a string", () => fresh() && h.b.write(5)); +t("write unknown", () => fresh() && h.b.write("ab", 0, "nope")); +t("write numeric string offset", () => fresh() && h.b.write("ab", "1")); +t("write fractional offset", () => fresh() && h.b.write("ab", 1.5)); +t("write offset past end", () => fresh() && h.b.write("ab", 12)); +t("write negative length", () => fresh() && h.b.write("ab", 0, -1)); +t("write number encoding", () => fresh() && h.b.write("ab", 0, 2, 5)); + +// fill with strings, numbers, buffers and other values. +t("fill string", () => fresh() && h.b.fill("ab", 0, 4)); +t("fill number", () => fresh() && h.b.fill(0x41, 4, 6)); +t("fill 300", () => fresh() && h.b.fill(300)); +t("fill true", () => fresh() && h.b.fill(true, 1, 3)); +t("fill object", () => fresh() && h.b.fill({}, 9)); +t("fill buffer", () => fresh() && h.b.fill(Buffer.from([1, 2]))); +t("fill encoding", () => fresh() && h.b.fill("ab", "utf16le")); +t("fill end encoding", () => fresh() && h.b.fill("ab", 1, "latin1")); +t("fill empty string", () => fresh() && h.b.fill("", 2)); +t("fill odd hex", () => fresh() && h.b.fill("abc", "hex")); +t("fill invalid hex", () => fresh() && h.b.fill("zz", "hex")); +t("fill empty buffer", () => fresh() && h.b.fill(Buffer.alloc(0))); +t("fill number encoding", () => fresh() && h.b.fill("a", 0, 1, 5)); +t("fill string offset", () => fresh() && h.b.fill(1, "2")); +t("fill end past length", () => fresh() && h.b.fill(1, 0, 20)); +t("fill offset past end", () => fresh() && h.b.fill(1, 20)); +t("fill empty range", () => fresh() && h.b.fill(1, 4, 2)); + +// toString windows and encodings. +t("toString window", () => fresh() && h.b.toString("hex", 1, 3)); +t("toString numeric strings", () => fresh() && h.b.toString("utf8", "1", "3")); +t("toString fractions", () => fresh() && h.b.toString("utf8", 0.9, 2.9)); +t("toString NaN", () => fresh() && h.b.toString("utf8", "x", "y")); +t("toString negative", () => fresh() && h.b.toString("utf8", -1, 2)); +t("toString undefined encoding", () => fresh() && h.b.toString(undefined, 6)); +t("toString empty unknown", () => fresh() && h.b.toString("nope", 3, 1)); +t("toString null encoding", () => fresh() && h.b.toString(null, 0, 1)); + +// The other Buffer methods. +t("toJSON", () => fresh() && h.b.toJSON()); +t("equals", () => fresh() && [h.b.equals(Buffer.from("hello world")), h.b.equals(Buffer.from("hello"))]); +t("equals string", () => fresh() && h.b.equals("hello world")); +t("compare", () => fresh() && [h.b.compare(Buffer.from("hellp")), h.b.compare(Buffer.from("xello"), 1, 5, 1, 5)]); +t("compare string", () => fresh() && h.b.compare("hello")); +t("copy", () => { + const d = Buffer.alloc(4); + return fresh() && [h.b.copy(d, 1, 6), d.toString()]; +}); +t("copy coerced offsets", () => { + const d = Buffer.alloc(4); + return fresh() && [h.b.copy(d, 1.7, "1", 3), d.toString("hex")]; +}); +t("copy NaN offsets", () => fresh() && h.b.copy(Buffer.alloc(2), NaN, "x")); +t("copy into Uint8Array", () => { + const d = new Uint8Array(3); + return fresh() && [h.b.copy(d), Array.from(d)]; +}); +t("copy negative target start", () => fresh() && h.b.copy(Buffer.alloc(2), -1)); +t("copy source start past end", () => fresh() && h.b.copy(Buffer.alloc(2), 0, 20)); +t("copy into array", () => fresh() && h.b.copy([1, 2])); +t("indexOf", () => fresh() && [h.b.indexOf("o"), h.b.indexOf("o", 5), h.b.indexOf("o", -4), h.b.indexOf(0x77)]); +t("indexOf buffer", () => fresh() && h.b.indexOf(Buffer.from("wor"))); +t("indexOf encoding", () => fresh() && [h.b.indexOf("6c6c", "hex"), h.b.indexOf("l", 0, "latin1")]); +t("indexOf utf16le", () => Buffer.from("abcabc", "utf16le").indexOf("b", 0, "utf16le")); +t("indexOf unknown", () => fresh() && h.b.indexOf("l", 0, "nope")); +t("indexOf number ignores encoding", () => fresh() && h.b.indexOf(0x6c, 0, "nope")); +t("indexOf object", () => fresh() && h.b.indexOf({})); +t("lastIndexOf", () => fresh() && [h.b.lastIndexOf("o"), h.b.lastIndexOf("o", 5), h.b.lastIndexOf("l", "latin1")]); +t("lastIndexOf string offset", () => fresh() && h.b.lastIndexOf("o", "x")); +t("includes", () => fresh() && [h.b.includes("wor"), h.b.includes("z"), h.b.includes("h", null)]); +t("swap16", () => { + h.b = Buffer.from("abcd"); + return h.b.swap16(); +}); +t("swap32 odd", () => { + h.b = Buffer.from("abc"); + return h.b.swap32(); +}); +t("readUInt16LE", () => fresh() && h.b.readUInt16LE(1)); + +// A buffer assigned to a constructor-function instance after construction. +function Later() {} +const later = new Later(); +later.buf = Buffer.alloc(4); +later.buf.write("cd", 1, "ascii"); +console.log("later", later.buf.toString("hex"), later.buf.fill(7).toString("hex")); + +// The same method names on receivers that are not Buffers. +function Box() { + this.text = "banana"; + this.list = [1, 2, 3]; + this.bytes = new Uint8Array([5, 6, 7]); + this.sink = { written: [], write(s) { this.written.push(s); return true; }, toString(a, b) { return `sink ${a} ${b}`; } }; +} +const box = new Box(); +console.log(box.text.indexOf("an"), box.text.lastIndexOf("an"), box.text.includes("nan"), box.text.toString()); +console.log(box.list.indexOf(2), box.list.includes(4), box.list.fill(0, 1).join(","), box.list.toString()); +console.log(Array.from(box.bytes.fill(9, 2)), box.bytes.toString()); +console.log(box.sink.write("x"), box.sink.written, box.sink.toString(1, 2)); diff --git a/tests/harness/corpus-costs/6930-dyn-buffer-methods.cjs.json b/tests/harness/corpus-costs/6930-dyn-buffer-methods.cjs.json new file mode 100644 index 000000000..83181fc36 --- /dev/null +++ b/tests/harness/corpus-costs/6930-dyn-buffer-methods.cjs.json @@ -0,0 +1 @@ +{ "plain": 7800, "san": 39000 } From f168402e1f5c97f517101543bf08eefc49bfa3fa Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 08:52:18 -0500 Subject: [PATCH 11/51] Name the reason in the CompileError for untranslated WebAssembly modules A WebAssembly module the build finds but does not translate (one that uses a proposal outside the translated feature set, an invalid one, or one the translator rejects) now rejects with a CompileError that says the module was found and names the reason, for example "it uses simd, outside the translated WebAssembly feature set". Bytes the build did not see keep the existing message. The build emits the hash, length and reason of each such module in a second table, scr_wasm_untranslated, next to scr_wasm_registry; the runtime reads it when a lookup misses. WebAssembly.validate still answers false for these modules, and the error stays a catchable CompileError. Fixes #1617 Co-authored-by: jlucaso1 <55464917+jlucaso1@users.noreply.github.com> --- .../limitations/webassembly-modules.mdx | 3 +- .../compiler/src/backend/llvm/wasm-modules.ts | 58 ++++++++--- .../compiler/src/frontend/lowering/lowerer.ts | 18 +++- .../src/ir/function-references.test.ts | 2 + packages/compiler/src/ir/ir.ts | 14 +++ packages/compiler/src/wasm/translate.ts | 4 +- packages/runtime/src/scr_wasm.c | 41 ++++++-- packages/runtime/src/scr_wasm_abi.h | 16 +++ .../harness/webassembly-untranslated.test.ts | 97 +++++++++++++++++++ 9 files changed, 228 insertions(+), 25 deletions(-) create mode 100644 tests/harness/webassembly-untranslated.test.ts diff --git a/docs/content/limitations/webassembly-modules.mdx b/docs/content/limitations/webassembly-modules.mdx index 43fa55e48..b74f2094d 100644 --- a/docs/content/limitations/webassembly-modules.mdx +++ b/docs/content/limitations/webassembly-modules.mdx @@ -5,7 +5,8 @@ Bytes the build did not see, including bytes produced at run time, reject with a `scriptc coverage` lists the modules a build finds, with their size, whether they validate, and the WebAssembly proposals they use. Translated modules can use the WebAssembly 2.0 feature set: the MVP plus sign-extension operators, non-trapping float-to-int conversions, bulk memory, multi-value, mutable globals, reference types, and extended constant expressions. -Modules that use SIMD, threads, exception handling, tail calls, typed function references or GC, memory64, or multiple memories are not translated, so loading them rejects with the `CompileError` above; packages that fall back to a scalar build when that happens, such as undici and es-module-lexer, take their fallback. +Modules that use SIMD, threads, exception handling, tail calls, typed function references or GC, memory64, or multiple memories are not translated, so loading them rejects with a `CompileError` that names the proposal the module uses, and `WebAssembly.validate` answers `false` for them; packages that fall back to a scalar build when that happens, such as undici and es-module-lexer, take their fallback. +Invalid modules the build finds reject with a `CompileError` that includes the validation error. Supported: `WebAssembly.compile`, `instantiate` (with bytes or a `Module`), `validate`, `new WebAssembly.Module`, `new WebAssembly.Instance`, `new WebAssembly.Memory`, `new WebAssembly.Table`, `new WebAssembly.Global`, `WebAssembly.Module.imports`, `exports` and `customSections`, and `instanceof` for every class; the error classes `CompileError`, `LinkError` and `RuntimeError`, constructed with or without `new`; `globalThis.WebAssembly` and its `compile`, `instantiate` and `validate` functions as values. Instances expose exported functions, memories (`buffer`, `grow`), tables (`get`, `set`, `grow`, `length`) and globals (`value`, `valueOf`). diff --git a/packages/compiler/src/backend/llvm/wasm-modules.ts b/packages/compiler/src/backend/llvm/wasm-modules.ts index 8a7045687..d08497214 100644 --- a/packages/compiler/src/backend/llvm/wasm-modules.ts +++ b/packages/compiler/src/backend/llvm/wasm-modules.ts @@ -1,12 +1,18 @@ /* The WebAssembly modules translated into a program (IrModule.wasmModules): - * each module's translation (wasm/translate.ts) and the registry the - * runtime looks modules up in (scr_wasm_registry, scr_wasm_abi.h). The + * each module's translation (wasm/translate.ts), the registry the runtime + * looks modules up in (scr_wasm_registry, scr_wasm_abi.h), and the table of + * modules the build saw but does not translate (scr_wasm_untranslated). The * translations join the program module, so the native helper optimizes, * partitions and caches them with it. Programs that do not use the * WebAssembly API carry no wasmModules and emit nothing here. */ -import type { IrModule } from "../../ir/ir.js"; +import type { IrModule, IrWasmUntranslated } from "../../ir/ir.js"; import { decodeBase64 } from "../../wasm/base64.js"; -import { translateWasmModule, WasmTranslateError } from "../../wasm/translate.js"; +import { + cBytes, + translateWasmModule, + utf8Bytes, + WasmTranslateError, +} from "../../wasm/translate.js"; function hexBytes(hex: string): Uint8Array { const out = new Uint8Array(hex.length / 2); @@ -28,11 +34,38 @@ export function wasmBoundsChecks(options: { return /windows|mingw|cygwin|win32|msvc/.test(target); } -/** The top-level lines of every translated module plus the registry. - * `declared` holds the declarations the module already carries; runtime - * entry points it lacks are declared here. A module the translator - * cannot translate is left out of the registry, so loading it at run - * time rejects with the CompileError for untranslated bytes. */ +/** scr_wasm_untranslated: hash, byte length and reason per module. */ +function emitUntranslated(untranslated: readonly IrWasmUntranslated[]): string[] { + if (untranslated.length === 0) + return [`@scr_wasm_untranslated = constant { i64, ptr } { i64 0, ptr null }`]; + const out: string[] = []; + const entries: string[] = []; + for (let index = 0; index < untranslated.length; index++) { + const module = untranslated[index]!; + const reason = new Uint8Array(utf8Bytes(module.reason)); + out.push( + `@sc_wasm_untranslated_reason${index} = private unnamed_addr constant [${reason.length} x i8] c"${cBytes(reason, 0, reason.length)}"`, + ); + const digest = hexBytes(module.sha256); + const sha: string[] = []; + for (let i = 0; i < 32; i++) sha.push(`i8 ${digest[i]!}`); + entries.push( + `{ [32 x i8], i64, ptr, i64 } { [32 x i8] [${sha.join(", ")}], i64 ${module.size}, ptr @sc_wasm_untranslated_reason${index}, i64 ${reason.length} }`, + ); + } + out.push( + `@sc_wasm_untranslated_modules = private constant [${entries.length} x { [32 x i8], i64, ptr, i64 }] [${entries.join(", ")}]`, + `@scr_wasm_untranslated = constant { i64, ptr } { i64 ${entries.length}, ptr @sc_wasm_untranslated_modules }`, + ); + return out; +} + +/** The top-level lines of every translated module plus the registry and + * the untranslated table. `declared` holds the declarations the module + * already carries; runtime entry points it lacks are declared here. A + * module the translator cannot translate joins the untranslated table, so + * loading it at run time rejects with a CompileError that names the + * translator's reason. */ export function emitWasmModules( mod: IrModule, declared: ReadonlySet, @@ -43,6 +76,7 @@ export function emitWasmModules( const out: string[] = []; const decls = new Set(declared); const descriptors: string[] = []; + const untranslated = [...(mod.wasmUntranslated ?? [])]; for (const module of modules) { const bytes = decodeBase64(module.bytes); if (bytes === null) continue; @@ -53,8 +87,9 @@ export function emitWasmModules( customSections: mod.wasmCustomSections === true, }); } catch (error) { - if (error instanceof WasmTranslateError) continue; - throw error; + if (!(error instanceof WasmTranslateError)) throw error; + untranslated.push({ sha256: module.sha256, size: bytes.length, reason: error.message }); + continue; } for (const decl of translation.decls) { if (decls.has(decl)) continue; @@ -72,6 +107,7 @@ export function emitWasmModules( `@scr_wasm_registry = constant { i64, ptr } { i64 ${descriptors.length}, ptr @sc_wasm_registry_modules }`, ); } + out.push(...emitUntranslated(untranslated)); out.push(""); return out; } diff --git a/packages/compiler/src/frontend/lowering/lowerer.ts b/packages/compiler/src/frontend/lowering/lowerer.ts index 543149f83..896f158a1 100644 --- a/packages/compiler/src/frontend/lowering/lowerer.ts +++ b/packages/compiler/src/frontend/lowering/lowerer.ts @@ -210,6 +210,7 @@ import type { IrLocal, IrModule, IrWasmModule, + IrWasmUntranslated, IrParam, IrRecordShape, IrStmt, @@ -1210,17 +1211,28 @@ export function lowerToIr( }); // A program that uses the WebAssembly API carries every module the build // can see and translate (wasm/translate.ts); the rest fail at run time - // with the CompileError for untranslated bytes. + // with a CompileError, which names the reason for the modules the build + // saw but does not translate. if (result.module !== null && resultLowerer.usesWebAssembly) { const bytesByHash = new Map(); const reports = discoverModules((sha256, bytes) => bytesByHash.set(sha256, bytes)); const wasmModules: IrWasmModule[] = []; + const wasmUntranslated: IrWasmUntranslated[] = []; for (const report of reports) { const bytes = bytesByHash.get(report.sha256); - if (report.status !== "valid" || bytes === undefined) continue; - wasmModules.push({ sha256: report.sha256, bytes: encodeBase64(bytes) }); + if (bytes === undefined) continue; + if (report.status === "valid") { + wasmModules.push({ sha256: report.sha256, bytes: encodeBase64(bytes) }); + continue; + } + const reason = + report.status === "unsupported" + ? `it uses ${report.unsupported.join(", ")}, outside the translated WebAssembly feature set` + : `it is not a valid module: ${report.error ?? report.status}`; + wasmUntranslated.push({ sha256: report.sha256, size: bytes.length, reason }); } result.module.wasmModules = wasmModules; + result.module.wasmUntranslated = wasmUntranslated; if (resultLowerer.usesWasmCustomSections) result.module.wasmCustomSections = true; timing("wasm-discovery", { modules: wasmModules.length }); } diff --git a/packages/compiler/src/ir/function-references.test.ts b/packages/compiler/src/ir/function-references.test.ts index 4ecc60a6f..7bef40aa6 100644 --- a/packages/compiler/src/ir/function-references.test.ts +++ b/packages/compiler/src/ir/function-references.test.ts @@ -214,6 +214,8 @@ const MODULE_FIELDS: Record>, Role> = { "contracts.programLifetime[]": "other", "wasmModules[].sha256": "other", "wasmModules[].bytes": "other", + "wasmUntranslated[].sha256": "other", + "wasmUntranslated[].reason": "other", "threadSlotKeys[]": "other", }; diff --git a/packages/compiler/src/ir/ir.ts b/packages/compiler/src/ir/ir.ts index 359d45ee7..603b9d0ae 100644 --- a/packages/compiler/src/ir/ir.ts +++ b/packages/compiler/src/ir/ir.ts @@ -1052,6 +1052,10 @@ export interface IrModule { * API (possibly empty). The backend appends each module's translation * and emits the registry the runtime looks modules up in. */ wasmModules?: IrWasmModule[]; + /** The modules the build saw but does not translate, with the reason, + * present with wasmModules. The backend emits them in a second table, so + * loading one rejects with a CompileError that names the reason. */ + wasmUntranslated?: IrWasmUntranslated[]; /** The program calls WebAssembly.Module.customSections, so translated * modules keep their custom sections (otherwise they carry none, and * debug sections cost nothing). */ @@ -1067,6 +1071,16 @@ export interface IrWasmModule { bytes: string; } +/** A WebAssembly module whose bytes the build saw but does not translate. */ +export interface IrWasmUntranslated { + /** Hex SHA-256 of the bytes. */ + sha256: string; + /** The byte length. */ + size: number; + /** Why the module is not translated, as a clause ("it uses simd, ..."). */ + reason: string; +} + /** Verified contract facts, keyed by IR names. */ export interface IrContracts { /** Int32 class field families: `${topmost declaring class}\0${field}`. */ diff --git a/packages/compiler/src/wasm/translate.ts b/packages/compiler/src/wasm/translate.ts index 4c02499ca..549b51a43 100644 --- a/packages/compiler/src/wasm/translate.ts +++ b/packages/compiler/src/wasm/translate.ts @@ -211,7 +211,7 @@ export function signatureHash(sig: string): string { } /** UTF-8 bytes of a string. */ -function utf8Bytes(text: string): number[] { +export function utf8Bytes(text: string): number[] { const out: number[] = []; for (let i = 0; i < text.length; i++) { const cp = text.codePointAt(i)!; @@ -233,7 +233,7 @@ function utf8Bytes(text: string): number[] { } /** An LLVM c"..." string body for bytes. */ -function cBytes(bytes: Uint8Array, start: number, end: number): string { +export function cBytes(bytes: Uint8Array, start: number, end: number): string { let s = ""; for (let i = start; i < end; i++) { const b = bytes[i]!; diff --git a/packages/runtime/src/scr_wasm.c b/packages/runtime/src/scr_wasm.c index 121acacbb..885f6adfe 100644 --- a/packages/runtime/src/scr_wasm.c +++ b/packages/runtime/src/scr_wasm.c @@ -5,7 +5,9 @@ * WebAssembly.compile, instantiate, validate and new Module hash the bytes * they are given and look the module up by SHA-256 and length in * scr_wasm_registry, which the compiler emits; bytes that were not - * translated at build time reject with a CompileError that says so. + * translated at build time reject with a CompileError that says so, and + * that names the reason for modules the build saw but does not translate + * (scr_wasm_untranslated). * Nothing is interpreted or compiled at run time, and a translated * module's bytes were validated when the program was built. * @@ -1668,12 +1670,35 @@ static const ScrWasmModuleDesc *scr_wasm_lookup(const uint8_t *data, size_t len) return NULL; } -static void scr_wasm_not_translated(const char *what) { - char msg[400]; - snprintf(msg, sizeof msg, - "%s: this module was not translated when the program was built (scriptc compiles WebAssembly ahead of " - "time and cannot compile modules whose bytes it did not see; see https://scriptc.dev/limitations)", - what); +/* The CompileError for bytes the registry does not hold: the reason the + * build recorded for a module it saw but does not translate, or the + * message for bytes it did not see. */ +static void scr_wasm_not_translated(const char *what, const uint8_t *data, size_t len) { + const ScrWasmUntranslatedDesc *found = NULL; + uint8_t digest[32]; + bool hashed = false; + for (uint64_t i = 0; i < scr_wasm_untranslated.count && !found; i++) { + const ScrWasmUntranslatedDesc *d = &scr_wasm_untranslated.modules[i]; + if (d->byte_length != len) continue; + if (!hashed) { + scr_wasm_sha256(data, len, digest); + hashed = true; + } + if (!memcmp(d->sha256, digest, 32)) found = d; + } + char msg[800]; + if (found) { + int reason_len = found->reason_len > 600 ? 600 : (int)found->reason_len; + snprintf(msg, sizeof msg, + "%s: this module was found when the program was built but not translated (%.*s; see " + "https://scriptc.dev/limitations)", + what, reason_len, found->reason); + } else { + snprintf(msg, sizeof msg, + "%s: this module was not translated when the program was built (scriptc compiles WebAssembly ahead of " + "time and cannot compile modules whose bytes it did not see; see https://scriptc.dev/limitations)", + what); + } scr_wasm_throw_kind(SCR_WASM_COMPILE_ERROR, msg); } @@ -1683,7 +1708,7 @@ static ScrDyn *scr_wasm_module_from(ScrDyn *bytes, const char *what, const char if (!scr_wasm_bytes_of(bytes, what, expected, &data, &len)) return NULL; const ScrWasmModuleDesc *desc = scr_wasm_lookup(data, len); if (!desc) { - scr_wasm_not_translated(what); + scr_wasm_not_translated(what, data, len); return NULL; } ScrWasmObj *o = scr_wasm_obj_new(SCR_WASM_K_MODULE); diff --git a/packages/runtime/src/scr_wasm_abi.h b/packages/runtime/src/scr_wasm_abi.h index 083a2f77a..d90deabc3 100644 --- a/packages/runtime/src/scr_wasm_abi.h +++ b/packages/runtime/src/scr_wasm_abi.h @@ -174,6 +174,22 @@ typedef struct { } ScrWasmRegistry; extern const ScrWasmRegistry scr_wasm_registry; +/* The modules the build saw but does not translate, emitted next to the + * registry: unsupported proposals, invalid modules and translator + * limits. `reason` is a clause naming why (not NUL-terminated). */ +typedef struct { + uint8_t sha256[32]; + uint64_t byte_length; + const char *reason; + uint64_t reason_len; +} ScrWasmUntranslatedDesc; + +typedef struct { + uint64_t count; + const ScrWasmUntranslatedDesc *modules; +} ScrWasmUntranslatedTable; +extern const ScrWasmUntranslatedTable scr_wasm_untranslated; + /* Runtime objects the translated code reads at fixed offsets. */ typedef struct ScrWasmMemory { uint8_t *base; /* 0 */ diff --git a/tests/harness/webassembly-untranslated.test.ts b/tests/harness/webassembly-untranslated.test.ts new file mode 100644 index 000000000..a12c75a19 --- /dev/null +++ b/tests/harness/webassembly-untranslated.test.ts @@ -0,0 +1,97 @@ +/* The CompileError for WebAssembly bytes that have no translation — + * scriptc-only. + * + * A module the build finds but does not translate (one that uses a + * proposal outside the translated feature set, or an invalid one) rejects + * with a CompileError that names the reason the build recorded. Bytes the + * build did not see reject with the message for unseen bytes. Both stay + * catchable CompileErrors, and WebAssembly.validate answers false for + * them. + */ +import { execFile } from "node:child_process"; +import { createHash } from "node:crypto"; +import { mkdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { describe, expect, test } from "vitest"; +import { compile } from "@scriptc/compiler"; + +const execFileAsync = promisify(execFile); +const repoRoot = join(import.meta.dirname, "../.."); +const cacheDir = join(repoRoot, "node_modules/.cache/scriptc-tests"); +const sanitize = process.env["SCRIPTC_SAN"] === "1"; + +// (module (func i32.const 0 i8x16.splat drop)) +const simdModule = "AGFzbQEAAAABBAFgAAADAgEACgkBBwBBAP0PGgs="; + +const source = `const simd = Buffer.from("${simdModule}", "base64"); +// A type section whose entry is not a function type, so the module is +// malformed. +const invalid = new Uint8Array([0, 97, 115, 109, 1, 0, 0, 0, 1, 2, 1, 64]); +// A custom section whose last byte is computed when the program runs, so +// the build does not see these bytes. +const unseen = Uint8Array.from([0, 97, 115, 109, 1, 0, 0, 0, 0, 4, 2, 97, 98, process.argv.length]); +function load(label, bytes) { + console.log(label, WebAssembly.validate(bytes)); + try { + new WebAssembly.Module(bytes); + } catch (e) { + console.log(label, e instanceof WebAssembly.CompileError, e.message); + } +} +load("simd", simd); +load("invalid", invalid); +load("unseen", unseen); +WebAssembly.compile(simd).catch((e) => console.log("compile", e instanceof WebAssembly.CompileError, e.message)); +`; + +async function run(): Promise { + const key = createHash("sha256") + .update(source) + .update(sanitize ? "san" : "plain") + .digest("hex") + .slice(0, 16); + const outDir = join(cacheDir, `wasm-untranslated-${key}`); + mkdirSync(outDir, { recursive: true }); + const file = join(outDir, "untranslated.mjs"); + writeFileSync(file, source); + const result = await compile(file, { + outPath: join(outDir, "untranslated"), + outDir, + sanitize, + backend: "llvm", + }); + if (!result.ok) { + throw new Error( + "program failed to compile:\n" + + result.diagnostics.map((d) => `${d.code}: ${d.message}`).join("\n"), + ); + } + const { stdout } = await execFileAsync(result.binaryPath, [], { encoding: "utf8" }); + return stdout; +} + +describe(`WebAssembly modules without a translation${sanitize ? " (sanitized)" : ""}`, () => { + test("the CompileError names why a module the build found is not translated", async () => { + const lines = (await run()).trimEnd().split("\n"); + const found = (what: string, reason: string) => + `${what}: this module was found when the program was built but not translated (${reason}; see https://scriptc.dev/limitations)`; + const unseen = + "WebAssembly.Module(): this module was not translated when the program was built (scriptc compiles " + + "WebAssembly ahead of time and cannot compile modules whose bytes it did not see; see https://scriptc.dev/limitations)"; + expect(lines[0]).toBe("simd false"); + expect(lines[1]).toBe( + `simd true ${found("WebAssembly.Module()", "it uses simd, outside the translated WebAssembly feature set")}`, + ); + expect(lines[2]).toBe("invalid false"); + expect(lines[3]).toBe( + `invalid true ${found("WebAssembly.Module()", "it is not a valid module: malformed function type (at byte 12)")}`, + ); + expect(lines[4]).toBe("unseen false"); + expect(lines[5]).toBe(`unseen true ${unseen}`); + expect(lines[6]).toBe( + `compile true ${found("WebAssembly.compile()", "it uses simd, outside the translated WebAssembly feature set")}`, + ); + expect(lines).toHaveLength(7); + }, 180_000); +}); From 927504af879f7f3ce8bbcc5fe83474450deef4e4 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 08:52:43 -0500 Subject: [PATCH 12/51] Read cached LLVM in parts on warm builds A warm build that finds the program's LLVM in the cache read the file into one string before emitting the program object. A program whose LLVM is past the engine's string limit (three.js nodes, 816 MB) failed there with `RangeError: Invalid string length`, though its cold build, which keeps the LLVM in parts, succeeds. The cache hit now reads the file with readTextParts: parts of at most 64 Mi UTF-16 units, decoded across reads, none ending inside a surrogate pair, which the emitter takes as a cold build hands them over. --- packages/compiler/src/index.ts | 6 ++- packages/compiler/src/text-parts.test.ts | 49 ++++++++++++++++++++++++ packages/compiler/src/text-parts.ts | 46 ++++++++++++++++++++++ 3 files changed, 99 insertions(+), 2 deletions(-) create mode 100644 packages/compiler/src/text-parts.test.ts diff --git a/packages/compiler/src/index.ts b/packages/compiler/src/index.ts index e9b8514af..66658fd45 100644 --- a/packages/compiler/src/index.ts +++ b/packages/compiler/src/index.ts @@ -112,7 +112,7 @@ import { import { type IrModule, type SrcLoc } from "./ir/ir.js"; import { moduleRuntimeFeatures } from "./ir/runtime-features.js"; import { serializeModuleParts } from "./ir/serialize.js"; -import { writeTextParts } from "./text-parts.js"; +import { readTextParts, writeTextParts } from "./text-parts.js"; import { validateModule } from "./ir/validate.js"; import { loadProgram } from "./frontend/program-node.js"; import { runFrontend, type FrontendFactory } from "./frontend/pipeline.js"; @@ -1197,7 +1197,9 @@ async function compileTracked( nativeProgramObject = await emitNativeProgramObject( entryPath, opts, - await readFile(earlyHit.llvmPath, "utf8"), + // A large program's LLVM exceeds the engine's string limit as one + // string; the emitter takes it in parts, as on a cold build. + await readTextParts(earlyHit.llvmPath), earlyHit.native, profile, ); diff --git a/packages/compiler/src/text-parts.test.ts b/packages/compiler/src/text-parts.test.ts new file mode 100644 index 000000000..4a09e13d5 --- /dev/null +++ b/packages/compiler/src/text-parts.test.ts @@ -0,0 +1,49 @@ +import { constants } from "node:buffer"; +import { mkdtempSync, rmSync, truncateSync, writeFileSync } from "node:fs"; +import { readFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterAll, expect, test } from "vitest"; +import { readTextParts, writeTextParts } from "./text-parts.js"; + +const scratch = mkdtempSync(join(tmpdir(), "scriptc-text-parts-")); +afterAll(() => rmSync(scratch, { recursive: true, force: true })); + +test("parts join to the file's text, each within the part length", async () => { + // Multi-byte sequences and surrogate pairs straddle the 1 MiB reads and + // the 8-unit parts. + const unit = "abéc中\u{1f600}d"; + const text = unit.repeat(90_000); + const path = join(scratch, "mixed.ll"); + writeFileSync(path, text); + const parts = await readTextParts(path, 8); + expect(parts.join("")).toBe(text); + for (const part of parts) { + expect(part.length).toBeLessThanOrEqual(8); + const last = part.charCodeAt(part.length - 1); + expect(last >= 0xd800 && last <= 0xdbff).toBe(false); + } + // Each part encodes on its own, so writing them back reproduces the file. + const copy = join(scratch, "copy.ll"); + await writeTextParts(copy, parts); + expect(await readFile(copy, "utf8")).toBe(text); +}); + +test("an empty file has no parts", async () => { + const path = join(scratch, "empty.ll"); + writeFileSync(path, ""); + expect(await readTextParts(path)).toEqual([]); +}); + +test("a file past the engine's string limit reads in parts", async () => { + // A sparse file: its bytes are NUL characters, valid UTF-8, and cost no + // disk space. + const path = join(scratch, "huge.ll"); + const size = constants.MAX_STRING_LENGTH + 1; + writeFileSync(path, ""); + truncateSync(path, size); + await expect(readFile(path, "utf8")).rejects.toThrow(RangeError); + const parts = await readTextParts(path); + expect(parts.length).toBeGreaterThan(1); + expect(parts.reduce((length, part) => length + part.length, 0)).toBe(size); +}, 120_000); diff --git a/packages/compiler/src/text-parts.ts b/packages/compiler/src/text-parts.ts index a4519b0ba..1b48e72fc 100644 --- a/packages/compiler/src/text-parts.ts +++ b/packages/compiler/src/text-parts.ts @@ -1,4 +1,10 @@ +import { createReadStream } from "node:fs"; import { open } from "node:fs/promises"; +import { StringDecoder } from "node:string_decoder"; + +/** The longest part readTextParts returns, far below the engine's string + * limit. */ +const TEXT_PART_LENGTH = 64 * 1024 * 1024; /** Writes compiler text artifacts (LLVM, serialized IR) in large batches. * Programs past the engine's string limit arrive as many parts, and writing @@ -35,3 +41,43 @@ export async function writeTextParts( await file.close(); } } + +/** Reads a text artifact as parts of at most `partLength` UTF-16 units: + * cached LLVM and serialized IR of large programs exceed the engine's + * string limit as one string. The decoder carries a multi-byte sequence + * split across reads into the next part, and no part ends inside a + * surrogate pair, so the parts join to the file's text and each encodes on + * its own. */ +export async function readTextParts( + path: string, + partLength = TEXT_PART_LENGTH, +): Promise { + const decoder = new StringDecoder("utf8"); + const parts: string[] = []; + let pending: string[] = []; + let pendingLength = 0; + const flush = (): void => { + if (pendingLength === 0) return; + parts.push(pending.join("")); + pending = []; + pendingLength = 0; + }; + const add = (text: string): void => { + for (let offset = 0; offset < text.length;) { + let take = Math.min(text.length - offset, partLength - pendingLength); + // A part never ends inside a surrogate pair: writeTextParts encodes + // its batches separately. + const last = text.charCodeAt(offset + take - 1); + if (take > 1 && last >= 0xd800 && last <= 0xdbff && offset + take < text.length) take--; + pending.push(text.slice(offset, offset + take)); + pendingLength += take; + offset += take; + if (pendingLength >= partLength - 1) flush(); + } + }; + for await (const chunk of createReadStream(path, { highWaterMark: 1 << 20 })) + add(decoder.write(chunk as Buffer)); + add(decoder.end()); + flush(); + return parts; +} From 9c1cabb08cde80966b0af87f30c3f88f23dd39d2 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 08:57:32 -0500 Subject: [PATCH 13/51] Remove repeated lookups from the class dynamic-dispatch passes ClassDynamicDispatch.process runs once per reachability wave and builds dispatch arms for every boxed class. Several lookups inside it grew with the program and repeated for every dispatch or member: - propertyReceivers was recomputed at every call site, including once per property dispatch. It now returns its last plans for a mode while the boxed classes, the registered classes, the subclass links (Lowerer.subclassLinks) and the constructed classes (Lowerer.constructedClassCount) are unchanged, and each boxed class still maps to the same ClassInfo. In canonical mode it also skips the subtree of a class it has already seen, which adds no plans. - The property dispatch loop tests the answered-class set before the member lookups, and the method and protocol loops test candidates in a Set instead of with Array.includes. A dynamic method call scans the boxed constructors for a static method only when no instance candidate answers the method, the only case that reads the result. - Lowerer.liftedFnNamed indexes liftedFns by name and replaces the linear liftedFns.find scans in the dispatch, prototype, descriptor and method-value helpers. The emitted IR is unchanged. --- .../frontend/lowering/class-descriptors.ts | 2 +- .../lowering/class-dynamic-dispatch.test.ts | 3 + .../lowering/class-dynamic-dispatch.ts | 71 ++++++++++++++----- .../frontend/lowering/class-method-values.ts | 2 +- .../src/frontend/lowering/class-prototypes.ts | 4 +- .../src/frontend/lowering/lower-classes.ts | 15 +++- .../compiler/src/frontend/lowering/lowerer.ts | 22 ++++++ 7 files changed, 94 insertions(+), 25 deletions(-) diff --git a/packages/compiler/src/frontend/lowering/class-descriptors.ts b/packages/compiler/src/frontend/lowering/class-descriptors.ts index cb980f0d0..0b807e450 100644 --- a/packages/compiler/src/frontend/lowering/class-descriptors.ts +++ b/packages/compiler/src/frontend/lowering/class-descriptors.ts @@ -177,7 +177,7 @@ export function lowerClassDataDescriptor( function descriptorGuard(lowerer: Lowerer, info: ClassInfo, loc: SrcLoc): string { const name = `%descriptorGuard:${info.def.name}`; - const existing = lowerer.liftedFns.find((fn) => fn.name === name); + const existing = lowerer.liftedFnNamed(name); const key = varRef("key", DYN, loc); const fields = new Set(); const collect = (owner: ClassInfo): void => { diff --git a/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.test.ts b/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.test.ts index 1c5c84cdf..dfb8bd7a5 100644 --- a/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.test.ts +++ b/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.test.ts @@ -32,6 +32,9 @@ function context() { unions: new Map(), classes: new Map(), liftedFns, + liftedFnNamed: (name: string | undefined) => liftedFns.find((fn) => fn.name === name), + subclassLinks: 0, + constructedClassCount: () => -1, diags: [], prototypeMethodAccesses: new Map(), protocolDispatches: new Map(), diff --git a/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts b/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts index cb5c28444..d9ef0de62 100644 --- a/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts +++ b/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts @@ -366,7 +366,7 @@ function readOnlyBagBody(body: IrStmt[], orEmpty = false): IrStmt[] { export function classPropertiesHelper(lowerer: Lowerer, loc: SrcLoc, readOnly = false): IrFunction { const name = readOnly ? READ_PROPERTIES : "%dyn.class.properties"; - const existing = lowerer.liftedFns.find((fn) => fn.name === name); + const existing = lowerer.liftedFnNamed(name); if (existing) return existing; const helper: IrFunction = { name, @@ -1003,7 +1003,7 @@ export class ClassDynamicDispatch { const loc = this.propertyBag.loc; const method = classMethodValue(lowerer, info.decl, info, "toJSON", loc); const prototype = method ? classPrototypeData(lowerer, info, loc) : null; - const helper = lowerer.liftedFns.find((fn) => fn.name === info.def.prototypeDataHelper); + const helper = lowerer.liftedFnNamed(info.def.prototypeDataHelper); const init = helper?.body[0]; if ( !info.prototypeReflectionReady && @@ -1236,6 +1236,15 @@ export class ClassDynamicDispatch { byMethod.set(method, matching); return matching; }; + const candidateSets = new Map>(); + const candidateSet = (method: string): ReadonlySet => { + let set = candidateSets.get(method); + if (!set) { + set = new Set(candidates(method)); + candidateSets.set(method, set); + } + return set; + }; for (const fn of functions) { // Discovery already visits every expression. Preserve bodies without // dispatch sites instead of rebuilding their entire typed IR tree on @@ -2220,16 +2229,16 @@ export class ClassDynamicDispatch { : expr.kind === "dynInvoke" ? expr : null; - const staticCandidates = - invoke && + // Static methods matter only for methods no instance answers. + const staticCandidates = (method: string): boolean => [...this.boxedConstructors].some((name) => { const info = lowerer.classes.get(name); - return info && findStaticOn(lowerer, info, invoke.method)?.method; + return info && findStaticOn(lowerer, info, method)?.method; }); if ( !invoke || (candidates(invoke.method).length === 0 && - !staticCandidates && + !staticCandidates(invoke.method) && !(invoke.method === "toString" && lowerer.classes.has("%Error"))) ) return expr; @@ -2501,10 +2510,13 @@ export class ClassDynamicDispatch { }); changed = true; } - for (const plan of this.propertyReceivers(lowerer, true)) { + // Compute the candidates where the first plan would have, so the + // memoized set is the same whichever plans are already answered. + const plans = this.propertyReceivers(lowerer, true); + const matching = plans.length ? candidateSet(dispatch.source.method) : null; + for (const plan of plans) { const { info } = plan; - if (!candidates(dispatch.source.method).includes(info) || dispatch.classes.has(plan.key)) - continue; + if (dispatch.classes.has(plan.key) || !matching!.has(info)) continue; dispatch.classes.add(plan.key); const loc = dispatch.source.loc; const value = varRef("p.0", DYN, loc); @@ -2648,7 +2660,7 @@ export class ClassDynamicDispatch { if ( plan.info !== plan.capsule || dispatch.classes.has(plan.key) || - (!dispatch.field && !candidates(dispatch.method).includes(plan.info)) + (!dispatch.field && !candidateSet(dispatch.method).has(plan.info)) ) continue; dispatch.classes.add(plan.key); @@ -2738,6 +2750,8 @@ export class ClassDynamicDispatch { } for (const plan of this.propertyReceivers(lowerer, true)) { const { info } = plan; + // Classes answered in an earlier pass skip the member lookups. + if (dispatch.classes.has(plan.key)) continue; if ( !isClassOwnEnumerableFieldName(dispatch.name) || (!(info.builtinStream !== undefined && streamProperty(dispatch.name)) && @@ -2751,7 +2765,6 @@ export class ClassDynamicDispatch { !findMethodOn(lowerer, info, `set:${dispatch.name}`)) ) continue; - if (dispatch.classes.has(plan.key)) continue; dispatch.classes.add(plan.key); const loc = dispatch.fn.loc; const { receiver, condition } = this.propertyReceiver(plan, loc); @@ -3310,7 +3323,7 @@ export class ClassDynamicDispatch { loc: SrcLoc, ): IrExpr { const name = fn === "dyn.hasKey" ? "%dyn.class.bagHas" : "%dyn.class.bagHasComputed"; - let helper = lowerer.liftedFns.find((item) => item.name === name); + let helper = lowerer.liftedFnNamed(name); if (!helper) { const keyType = fn === "dyn.hasKey" ? STRING : DYN; const value = varRef("p.0", DYN, loc); @@ -3907,7 +3920,7 @@ export class ClassDynamicDispatch { private readPropertyBagOrEmpty(lowerer: Lowerer, receiver: IrExpr, loc: SrcLoc): IrExpr { // One call per read site: large JS programs have a branch per class and // property, so the absent-bag handling lives in the shared helper. - let helper = lowerer.liftedFns.find((fn) => fn.name === READ_PROPERTIES_OR_EMPTY); + let helper = lowerer.liftedFnNamed(READ_PROPERTIES_OR_EMPTY); if (!helper) { // Settled by finalize; until then, the read-only bag's own dispatch. helper = { @@ -4256,17 +4269,38 @@ export class ClassDynamicDispatch { add(root); } + /** propertyReceivers results by mode, with the class graph they were + * computed from: the boxed classes, the registered classes, the subclass + * links and the constructed classes only grow, so equal counts and the + * same capsules give the same plans. */ + private readonly receiverPlans = new Map< + string, + { revision: string; capsules: (ClassInfo | undefined)[]; plans: readonly PropertyReceiver[] } + >(); + /** A capsule retains its static type, while its object may be a subclass. * Preorder insertion lets later branches test the most derived layout first. */ private propertyReceivers( lowerer: Lowerer, streams = false, normalize = true, - ): PropertyReceiver[] { + ): readonly PropertyReceiver[] { + const mode = `${streams}:${normalize}`; + const revision = `${this.boxed.size}:${lowerer.classes.size}:${lowerer.subclassLinks}:${lowerer.constructedClassCount()}`; + const cached = this.receiverPlans.get(mode); + if (cached?.revision === revision) { + let i = 0; + let same = true; + for (const name of this.boxed) + if (lowerer.classes.get(name) !== cached.capsules[i++]) same = false; + if (same) return cached.plans; + } + const capsules: (ClassInfo | undefined)[] = []; const plans: PropertyReceiver[] = []; const seen = new Set(); for (const name of this.boxed) { const capsule = lowerer.classes.get(name); + capsules.push(capsule); if ( !capsule || ((capsule.builtinEmitter || capsule.builtinStream) && !streams) || @@ -4275,10 +4309,10 @@ export class ClassDynamicDispatch { continue; const canonical = normalize && !capsule.builtinStream && !capsule.builtinEmitter; const visit = (info: ClassInfo): void => { - if ( - (!canonical || !seen.has(info.def.name)) && - (info === capsule || lowerer.classCanBeConstructed(info)) - ) { + // A seen class was visited with its whole subtree, and a canonical + // visit adds no plan for a seen class. + if (canonical && seen.has(info.def.name)) return; + if (info === capsule || lowerer.classCanBeConstructed(info)) { seen.add(info.def.name); plans.push({ info, @@ -4290,6 +4324,7 @@ export class ClassDynamicDispatch { }; visit(capsule); } + this.receiverPlans.set(mode, { revision, capsules, plans }); return plans; } diff --git a/packages/compiler/src/frontend/lowering/class-method-values.ts b/packages/compiler/src/frontend/lowering/class-method-values.ts index 1197d9bdd..6e8256513 100644 --- a/packages/compiler/src/frontend/lowering/class-method-values.ts +++ b/packages/compiler/src/frontend/lowering/class-method-values.ts @@ -426,7 +426,7 @@ function reflectedMethodValue( ); } if (value.kind === "closure" && value.captures.length === 0) { - const body = lowerer.liftedFns.find((fn) => fn.name === value.fnName); + const body = lowerer.liftedFnNamed(value.fnName); if (body?.captures?.length === 0) delete body.captures; } const cache = staticMethod diff --git a/packages/compiler/src/frontend/lowering/class-prototypes.ts b/packages/compiler/src/frontend/lowering/class-prototypes.ts index 7b4b58ae3..d2919e202 100644 --- a/packages/compiler/src/frontend/lowering/class-prototypes.ts +++ b/packages/compiler/src/frontend/lowering/class-prototypes.ts @@ -172,7 +172,7 @@ export function classPrototypeData( } if (!metadataOnly && !info.prototypeObservedMethodsReady) { info.prototypeObservedMethodsReady = true; - const helper = lowerer.liftedFns.find((fn) => fn.name === info.def.prototypeDataHelper)!; + const helper = lowerer.liftedFnNamed(info.def.prototypeDataHelper)!; const value = varRef(`%g.${info.def.prototypeDataHelper}`, DYN, loc); // Materialize only observed method slots. Own declarations stop lookup // at the correct prototype even if an ancestor is replaced later. An @@ -307,7 +307,7 @@ export function reflectClassPrototype( if (info.base.prototypeReflectionRefusal) return refuse("prototype reflection with this base class is not supported yet"); } - const helper = lowerer.liftedFns.find((fn) => fn.name === info.def.prototypeDataHelper); + const helper = lowerer.liftedFnNamed(info.def.prototypeDataHelper); const initialization = helper?.body[0]; if (!helper || initialization?.kind !== "if" || info.prototypeReflectionReady) return prototype; info.prototypeReflectionReady = true; diff --git a/packages/compiler/src/frontend/lowering/lower-classes.ts b/packages/compiler/src/frontend/lowering/lower-classes.ts index 371541df8..f312e9229 100644 --- a/packages/compiler/src/frontend/lowering/lower-classes.ts +++ b/packages/compiler/src/frontend/lowering/lower-classes.ts @@ -946,7 +946,10 @@ export function registerBuiltinErrorClasses(lowerer: Lowerer): void { throwingSetters: [], staticFields: [], }; - if (base) base.subclasses.push(info); + if (base) { + base.subclasses.push(info); + lowerer.subclassLinks++; + } lowerer.classes.set(irName, info); } } @@ -1026,7 +1029,10 @@ export function registerBuiltinStreamClasses(lowerer: Lowerer): void { throwingSetters: [], staticFields: [], }; - if (base) base.subclasses.push(info); + if (base) { + base.subclasses.push(info); + lowerer.subclassLinks++; + } lowerer.classes.set(irName, info); } } @@ -4471,7 +4477,10 @@ export function collectClassShapeInner( }; lowerer.genericClassByDecl.set(decl, info.generic); } - if (base) base.subclasses.push(info); + if (base) { + base.subclasses.push(info); + lowerer.subclassLinks++; + } lowerer.classes.set(className, info); if (computedBase && !functionLocalClass(decl)) { info.def.baseValueGlobal = `%g.${className}.computedBase`; diff --git a/packages/compiler/src/frontend/lowering/lowerer.ts b/packages/compiler/src/frontend/lowering/lowerer.ts index 543149f83..be5db5cc6 100644 --- a/packages/compiler/src/frontend/lowering/lowerer.ts +++ b/packages/compiler/src/frontend/lowering/lowerer.ts @@ -3042,6 +3042,10 @@ export class Lowerer { // top level, top = the innermost nested function currently lowering). fnStack: FnCtx[] = []; readonly liftedFns: IrFunction[] = []; + /** The first lifted function of each name (liftedFnNamed), for the + * liftedFns indexed so far. liftedFns only grows. */ + private readonly liftedFnsByName = new Map(); + private liftedFnsIndexed = 0; /** Runtime-provided classes used as values (defineClassValueConstruct), * by IR class name. */ readonly runtimeClassValues = new Map(); @@ -3113,6 +3117,10 @@ export class Lowerer { * can contain any retained descendant, but importing its declaration * alone does not make every descendant constructible. */ private constructedClasses: Set | null = null; + /** Classes added to some ClassInfo's subclasses (lower-classes.ts). With + * the class and constructed-class counts, it tells caches over the class + * graph when the graph changed. */ + subclassLinks = 0; /** Set while a subclass constructor's super() call notes its base * constructor edge: that edge does not instantiate the base exactly. */ private superConstructorEdge = false; @@ -3129,6 +3137,20 @@ export class Lowerer { info.mixinInstance !== undefined ); } + /** How many classes classCanBeConstructed admits by name, or -1 while + * it admits every class. The count only grows. */ + constructedClassCount(): number { + return this.constructedClasses?.size ?? -1; + } + /** The first lifted function named `name`, the one + * `liftedFns.find((fn) => fn.name === name)` returns. */ + liftedFnNamed(name: string | undefined): IrFunction | undefined { + for (; this.liftedFnsIndexed < this.liftedFns.length; this.liftedFnsIndexed++) { + const fn = this.liftedFns[this.liftedFnsIndexed]!; + if (!this.liftedFnsByName.has(fn.name)) this.liftedFnsByName.set(fn.name, fn); + } + return name === undefined ? undefined : this.liftedFnsByName.get(name); + } /** Coverage remainder mode: the reachability gate inverts (see wantBody) * and no module is built. */ readonly remainder: boolean; From 9b70fb5e7b0fde8a6eeb87bb903e47c7b2d40b88 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 08:58:29 -0500 Subject: [PATCH 14/51] Negotiate ALPN and bound the TLS version in tls.connect and tls.createServer tls.connect and tls.createServer accept ALPNProtocols as an array of strings, a Buffer or a Uint8Array in the wire format. The server answers with the first of its own protocols that the client offers, and both sockets report it as alpnProtocol. A client that offers none of the server's protocols fails with ERR_SSL_TLSV1_ALERT_NO_APPLICATION_PROTOCOL. minVersion and maxVersion bound the handshake, and getProtocol() reports the version it picked. Ranges that do not overlap fail with Node's ERR_SSL_TLSV1_ALERT_PROTOCOL_VERSION or ERR_SSL_NO_PROTOCOLS_AVAILABLE, and invalid versions throw Node's TypeError. tls.connect's timeout option arms the socket's inactivity timeout before it connects. Refs #1438 Co-authored-by: saboursepideh <25549001+saboursepideh@users.noreply.github.com> --- .../src/generated/node-v24-compatibility.json | 12 +- .../generated/node-v24-internal.json | 18 +- internal/compatibility/static-support.json | 2 + .../src/frontend/lowering/lower-server.ts | 30 +- .../corpus/6721-tls-alpn-protocols.ts.json | 6 + .../corpus/6722-tls-min-max-version.ts.json | 6 + .../corpus/6723-tls-connect-timeout.ts.json | 6 + packages/runtime/src/scr_error.c | 5 + packages/runtime/src/scr_tls.c | 443 ++++++++++++++++-- tests/corpus/6721-tls-alpn-protocols.ts | 75 +++ tests/corpus/6722-tls-min-max-version.ts | 85 ++++ tests/corpus/6723-tls-connect-timeout.ts | 51 ++ .../6721-tls-alpn-protocols.ts.json | 1 + .../6722-tls-min-max-version.ts.json | 1 + .../6723-tls-connect-timeout.ts.json | 1 + 15 files changed, 694 insertions(+), 48 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6721-tls-alpn-protocols.ts.json create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6722-tls-min-max-version.ts.json create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6723-tls-connect-timeout.ts.json create mode 100644 tests/corpus/6721-tls-alpn-protocols.ts create mode 100644 tests/corpus/6722-tls-min-max-version.ts create mode 100644 tests/corpus/6723-tls-connect-timeout.ts create mode 100644 tests/harness/corpus-costs/6721-tls-alpn-protocols.ts.json create mode 100644 tests/harness/corpus-costs/6722-tls-min-max-version.ts.json create mode 100644 tests/harness/corpus-costs/6723-tls-connect-timeout.ts.json diff --git a/docs/src/generated/node-v24-compatibility.json b/docs/src/generated/node-v24-compatibility.json index ed717c35e..66b0ba3dd 100644 --- a/docs/src/generated/node-v24-compatibility.json +++ b/docs/src/generated/node-v24-compatibility.json @@ -68082,9 +68082,9 @@ "inherited": true }, "static": { - "status": "not-implemented", - "detail": "No static compiler lowering is registered for this API yet.", - "verification": "registry-gap" + "status": "partial", + "detail": "Implemented for the documented scriptc module-loader subset.", + "verification": "test-backed" } }, { @@ -68289,9 +68289,9 @@ "inherited": true }, "static": { - "status": "not-implemented", - "detail": "No static compiler lowering is registered for this API yet.", - "verification": "registry-gap" + "status": "partial", + "detail": "Implemented for the documented scriptc module-loader subset.", + "verification": "test-backed" } }, { diff --git a/internal/compatibility/generated/node-v24-internal.json b/internal/compatibility/generated/node-v24-internal.json index bbf177d52..c6a7a102c 100644 --- a/internal/compatibility/generated/node-v24-internal.json +++ b/internal/compatibility/generated/node-v24-internal.json @@ -95364,7 +95364,9 @@ "tests/corpus/2331-tls-server-options-typed.ts", "tests/corpus/2500-net-autosel-timeout.ts", "tests/corpus/2557-tls-ca-store.ts", - "tests/corpus/2596-net-arg-ladders.cjs" + "tests/corpus/2596-net-arg-ladders.cjs", + "tests/corpus/6721-tls-alpn-protocols.ts", + "tests/corpus/6722-tls-min-max-version.ts" ] }, "anchorSource": "chapter" @@ -96365,8 +96367,11 @@ "inherited": true }, "static": { - "status": "not-implemented", - "evidence": "compiler-unmatched:tls" + "status": "partial", + "evidence": "compiler-feature:tls.tlsSocket.getProtocol", + "tests": [ + "tests/corpus/6722-tls-min-max-version.ts" + ] }, "anchorSource": "exact" }, @@ -96572,8 +96577,11 @@ "inherited": true }, "static": { - "status": "not-implemented", - "evidence": "compiler-unmatched:tls" + "status": "partial", + "evidence": "compiler-feature:tls.alpnProtocol", + "tests": [ + "tests/corpus/6721-tls-alpn-protocols.ts" + ] }, "anchorSource": "ancestor" }, diff --git a/internal/compatibility/static-support.json b/internal/compatibility/static-support.json index 1f713630b..23c7a159b 100644 --- a/internal/compatibility/static-support.json +++ b/internal/compatibility/static-support.json @@ -365,6 +365,8 @@ { "chapter": "stream", "signatures": ["`writable` Type: {boolean}"], "status": "partial", "evidence": ["tests/corpus/2893-child-stdin-duplex.ts", "tests/corpus/2895-child-stdin-write-after-end.ts"] }, { "chapter": "stream", "signatures": ["Event: `'drain'`"], "status": "partial", "evidence": ["tests/corpus/2894-child-stdin-backpressure.ts"] }, { "chapter": "console", "signatures": ["Class: `Console`"], "status": "partial", "evidence": ["tests/corpus/4900-console-instances.ts", "tests/corpus/4901-console-instance-errors.ts"] }, + { "chapter": "tls", "signatures": ["`alpnProtocol` Type: {string|boolean|null}"], "status": "partial", "evidence": ["tests/corpus/6721-tls-alpn-protocols.ts"] }, + { "chapter": "tls", "signatures": ["`tlsSocket.getProtocol()`"], "status": "partial", "evidence": ["tests/corpus/6722-tls-min-max-version.ts"] }, { "chapter": "webstreams", "signatures": ["`readableStream.values([options])`"], "status": "partial", "evidence": ["tests/corpus/2862-web-readable-async-iteration.ts"] }, { "chapter": "webstreams", "signatures": ["`streamConsumers.text(stream)`", "`streamConsumers.json(stream)`", "`streamConsumers.buffer(stream)`"], "status": "partial", "evidence": ["tests/corpus/2629-stream-consumers.ts", "tests/corpus/2630-stream-consumers-errors.ts"] }, { "chapter": "webstreams", "signatures": ["`streamConsumers.arrayBuffer(stream)`"], "status": "partial", "evidence": ["tests/corpus/4875-stream-consumers-array-buffer.ts"] }, diff --git a/packages/compiler/src/frontend/lowering/lower-server.ts b/packages/compiler/src/frontend/lowering/lower-server.ts index f0da85bf8..4a7c00d84 100644 --- a/packages/compiler/src/frontend/lowering/lower-server.ts +++ b/packages/compiler/src/frontend/lowering/lower-server.ts @@ -4877,12 +4877,14 @@ const TLS_VALIDATED_OPTIONS: ReadonlySet = new Set([ ]); /** True when a literal options bag must ride the runtime walker: JS uses - * it for the typed TLS validation ladders, and HTTPS uses it for + * it for the typed TLS validation ladders, HTTPS uses it for * keepAliveTimeoutBuffer so the HTTP option is consumed without losing - * the literal's evaluation order. */ + * the literal's evaluation order, and `runtimeKeys` names options only the + * runtime walk implements (tls.createServer's ALPNProtocols). */ function tlsLiteralNeedsRuntimeWalk( node: ts.ObjectLiteralExpression, includeHttpTimeoutOptions = false, + runtimeKeys: ReadonlySet = new Set(), ): boolean { const jsSource = isJsSourceFile(ts.getSourceFileOfNode(node)); return node.properties.nodes.some( @@ -4890,19 +4892,29 @@ function tlsLiteralNeedsRuntimeWalk( (ts.isPropertyAssignment(p) || ts.isShorthandPropertyAssignment(p)) && (ts.isIdentifier(p.name) || ts.isStringLiteral(p.name)) && ((includeHttpTimeoutOptions && p.name.text === "keepAliveTimeoutBuffer") || + runtimeKeys.has(p.name.text) || (jsSource && TLS_VALIDATED_OPTIONS.has(p.name.text))), ); } +/** tls.createServer options the runtime walk implements and the literal + * cert/key lowering does not. */ +const TLS_SERVER_RUNTIME_OPTIONS: ReadonlySet = new Set([ + "ALPNProtocols", + "minVersion", + "maxVersion", +]); + function lowerTlsServerOptionsOrDyn( lowerer: Lowerer, node: ts.Expression, what: string, includeHttpTimeoutOptions = false, + runtimeKeys: ReadonlySet = new Set(), ): { cert: IrExpr; key: IrExpr; dyn?: undefined } | { dyn: IrExpr } { if ( ts.isObjectLiteralExpression(node) && - !tlsLiteralNeedsRuntimeWalk(node, includeHttpTimeoutOptions) + !tlsLiteralNeedsRuntimeWalk(node, includeHttpTimeoutOptions, runtimeKeys) ) { return lowerTlsServerOptions(lowerer, node, what); } @@ -4934,14 +4946,14 @@ function lowerTlsServerOptionsOrDyn( * connect(options[, cb]) and connect(port[, host][, options][, cb]) with * a RUNTIME options record (the checked-dynamic JS lane's shape — a dyn * value, or a record that converts): the runtime walk implements - * port/host/rejectUnauthorized/ca/servername and throws the catchable + * port/host/rejectUnauthorized/ca/servername/ALPNProtocols/minVersion/maxVersion/timeout and throws the catchable * fence for other documented members (the divergence-66 stance). The * callback (and 'secureConnect'/'connect' listeners) fires * post-handshake — Node's secureConnect timing. */ function lowerTlsConnectCall(lowerer: Lowerer, expr: ts.CallExpression, loc: SrcLoc): IrExpr { const args = expr.arguments.nodes; const FORMS_HINT = - "the supported forms are connect(options[, cb]) and connect(port[, host][, options][, cb]) — options implement port/host/rejectUnauthorized/ca/servername"; + "the supported forms are connect(options[, cb]) and connect(port[, host][, options][, cb]) — options implement port/host/rejectUnauthorized/ca/servername/ALPNProtocols/minVersion/maxVersion/timeout"; if (args.length < 1 || args.length > 4 || args.some(ts.isSpreadElement)) { lowerer.noLowering(`tls.connect with ${args.length} arguments`, expr, FORMS_HINT); } @@ -5060,7 +5072,13 @@ function lowerTlsModuleCall( "the supported form is createServer({ cert, key }, socket => ...)", ); } - const opts = lowerTlsServerOptionsOrDyn(lowerer, args[0]!, "tls.createServer"); + const opts = lowerTlsServerOptionsOrDyn( + lowerer, + args[0]!, + "tls.createServer", + false, + TLS_SERVER_RUNTIME_OPTIONS, + ); if (args.length === 1) { return opts.dyn !== undefined ? { kind: "libCall", fn: "tls.createServerDyn", args: [opts.dyn], type: NETSERVER_T, loc } diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6721-tls-alpn-protocols.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6721-tls-alpn-protocols.ts.json new file mode 100644 index 000000000..b86035ce7 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6721-tls-alpn-protocols.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6721-tls-alpn-protocols.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6722-tls-min-max-version.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6722-tls-min-max-version.ts.json new file mode 100644 index 000000000..0100ffe8f --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6722-tls-min-max-version.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6722-tls-min-max-version.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6723-tls-connect-timeout.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6723-tls-connect-timeout.ts.json new file mode 100644 index 000000000..cc239efba --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6723-tls-connect-timeout.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6723-tls-connect-timeout.ts" + ], + "diags": [] +} diff --git a/packages/runtime/src/scr_error.c b/packages/runtime/src/scr_error.c index 4f514681b..94523fc43 100644 --- a/packages/runtime/src/scr_error.c +++ b/packages/runtime/src/scr_error.c @@ -422,6 +422,11 @@ const char *scr_error_message_code(const ScrStr *msg) { { "self-signed certificate", false, "DEPTH_ZERO_SELF_SIGNED_CERT" }, { "unable to verify the first certificate", false, "UNABLE_TO_VERIFY_LEAF_SIGNATURE" }, { "Hostname/IP does not match certificate's altnames", false, "ERR_TLS_CERT_ALTNAME_INVALID" }, + { "error:0A000460:SSL routines:ssl3_read_bytes:tlsv1 alert no application protocol", true, + "ERR_SSL_TLSV1_ALERT_NO_APPLICATION_PROTOCOL" }, + { "error:0A00042E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version", true, + "ERR_SSL_TLSV1_ALERT_PROTOCOL_VERSION" }, + { "error:0A0000BF:SSL routines::no protocols available", true, "ERR_SSL_NO_PROTOCOLS_AVAILABLE" }, }; for (size_t i = 0; i < sizeof known / sizeof known[0]; i++) { size_t n = strlen(known[i].prefix); diff --git a/packages/runtime/src/scr_tls.c b/packages/runtime/src/scr_tls.c index 804e51d69..59f3c2c0a 100644 --- a/packages/runtime/src/scr_tls.c +++ b/packages/runtime/src/scr_tls.c @@ -255,6 +255,93 @@ void scr_secure_ctx_release_v(void *p) { scr_secure_ctx_release((ScrSecureCtx *) /* ── the server config (one per created server, rc'd) ────────────────── */ +/* An owned ALPN name list (NULL-terminated; NULL frees nothing). */ +static void scr_tls_alpn_free(char **names) { + if (names == NULL) return; + for (char **n = names; *n != NULL; n++) free(*n); + free(names); +} + +static bool scr_tls_alpn_dyn(const ScrDyn *v, char ***out); +static void scr_tls_opt_fence(const char *api, const char *key, const char *hint); +static bool scr_tls_dyn_truthy(const ScrDyn *v); + +/* Node's toV over a minVersion/maxVersion value: 'TLSv1' through 'TLSv1.3' + * as 10 through 13, null or undefined as `def`. Anything else throws + * ERR_TLS_INVALID_PROTOCOL_VERSION ("%j is not a valid TLS protocol + * version"). False with the exception pending. */ +static bool scr_tls_version_dyn(const ScrDyn *v, const char *which, int def, int *out) { + if (v == NULL || v->kind == SCR_DYN_UNDEF || v->kind == SCR_DYN_NULL) { + *out = def; + return true; + } + if (v->kind == SCR_DYN_STR) { + static const struct { const char *name; int version; } names[] = { + { "TLSv1", 10 }, { "TLSv1.1", 11 }, { "TLSv1.2", 12 }, { "TLSv1.3", 13 }, + }; + for (size_t i = 0; i < sizeof names / sizeof names[0]; i++) { + if (v->v.str->len == strlen(names[i].name) && + memcmp(v->v.str->data, names[i].name, v->v.str->len) == 0) { + *out = names[i].version; + return true; + } + } + } + ScrStr *shown = scr_util_format_j(v); + if (shown == NULL) return false; + char msg[300]; + int m = snprintf(msg, sizeof msg, "%.*s is not a valid %s TLS protocol version", + (int)(shown->len < 200 ? shown->len : 200), shown->data, which); + scr_str_release(shown); + scr_throw_error_msg_code(SCR_ERR_TYPE, msg, (size_t)m, "ERR_TLS_INVALID_PROTOCOL_VERSION"); + return false; +} + +/* The version range on an mbedTLS config. mbedTLS implements TLS 1.2 and + * 1.3, so a minimum of TLSv1 or TLSv1.1 offers from 1.2, the lowest version + * Node's default security level allows too. Answers false when no version + * in [min, max] is available (a maximum below 1.2, or min above max), which + * Node reports as ERR_SSL_NO_PROTOCOLS_AVAILABLE when the handshake starts. */ +static bool scr_tls_conf_versions(mbedtls_ssl_config *conf, int min, int max) { + if (min < 12) min = 12; + if (max < min) return false; + mbedtls_ssl_conf_min_tls_version(conf, min == 13 ? MBEDTLS_SSL_VERSION_TLS1_3 : MBEDTLS_SSL_VERSION_TLS1_2); + mbedtls_ssl_conf_max_tls_version(conf, max == 13 ? MBEDTLS_SSL_VERSION_TLS1_3 : MBEDTLS_SSL_VERSION_TLS1_2); + return true; +} + +/* Node's SecureContext checks, in its order: secureProtocol conflicts with + * an explicit minVersion or maxVersion, then each version. The range lands + * in *min / *max (Node's defaults TLSv1.2 and TLSv1.3). False with the + * exception pending. */ +static bool scr_tls_versions_dyn(const ScrDyn *opts, int *min, int *max) { + const ScrDyn *vmin = opts != NULL && opts->kind == SCR_DYN_OBJ ? scr_dyn_obj_get(opts, "minVersion", 10) : NULL; + const ScrDyn *vmax = opts != NULL && opts->kind == SCR_DYN_OBJ ? scr_dyn_obj_get(opts, "maxVersion", 10) : NULL; + const ScrDyn *proto = opts != NULL && opts->kind == SCR_DYN_OBJ ? scr_dyn_obj_get(opts, "secureProtocol", 14) : NULL; + if (proto != NULL && scr_tls_dyn_truthy(proto)) { + const ScrDyn *version = vmin != NULL && vmin->kind != SCR_DYN_UNDEF && vmin->kind != SCR_DYN_NULL ? vmin + : vmax != NULL && vmax->kind != SCR_DYN_UNDEF && vmax->kind != SCR_DYN_NULL ? vmax + : NULL; + if (version != NULL) { + ScrStr *a = scr_util_format_j(version); + if (a == NULL) return false; + ScrStr *b = scr_util_format_j(proto); + if (b == NULL) { + scr_str_release(a); + return false; + } + char msg[400]; + int m = snprintf(msg, sizeof msg, "TLS protocol version %.*s conflicts with secureProtocol %.*s", + (int)(a->len < 150 ? a->len : 150), a->data, (int)(b->len < 150 ? b->len : 150), b->data); + scr_str_release(a); + scr_str_release(b); + scr_throw_error_msg_code(SCR_ERR_TYPE, msg, (size_t)m, "ERR_TLS_PROTOCOL_VERSION_CONFLICT"); + return false; + } + } + return scr_tls_version_dyn(vmin, "minimum", 12, min) && scr_tls_version_dyn(vmax, "maximum", 13, max); +} + typedef struct ScrTlsSrv { size_t rc; mbedtls_ssl_config conf; @@ -275,6 +362,12 @@ typedef struct ScrTlsSrv { * thunk that becomes each minted answer closure's fn */ ScrClosure *sni_cb; void *sni_answer_fn; + /* tls.createServer's ALPNProtocols (owned; NULL when none), in the + * server's preference order. */ + char **alpn; + /* maxVersion TLSv1.2: the ClientHello is pre-parsed so a hello that + * offers only later versions gets OpenSSL's protocol_version alert. */ + bool max_tls12; } ScrTlsSrv; /* The ALPN answers (static storage — mbedTLS keeps the pointers). Plain @@ -325,6 +418,7 @@ static void scr_tls_srv_release(ScrTlsSrv *s) { mbedtls_ssl_config_free(&s->conf); mbedtls_x509_crt_free(&s->cert); mbedtls_pk_free(&s->pk); + scr_tls_alpn_free(s->alpn); free(s); } @@ -342,6 +436,13 @@ typedef struct ScrTlsCli { * VERIFY_NONE socket (the https client's rejectUnauthorized:false) * never verified, so authorized answers false like Node's. */ bool verify_recorded; + /* The ALPNProtocols option's names, NULL-terminated, in the caller's + * order (mbedtls_ssl_conf_alpn_protocols keeps the pointer); NULL when + * none are offered. */ + char **alpn; + /* No TLS version is in the configured range: the handshake fails as + * soon as it starts (Node's ERR_SSL_NO_PROTOCOLS_AVAILABLE). */ + bool no_protocols; } ScrTlsCli; /* The default trust anchors when no `ca` option is given: the host trust @@ -490,6 +591,7 @@ static ScrTlsCli *scr_tls_cli_new(ScrStr *host, bool reject_unauthorized, const static void scr_tls_cli_free(ScrTlsCli *c) { mbedtls_ssl_config_free(&c->conf); + scr_tls_alpn_free(c->alpn); mbedtls_x509_crt_free(&c->ca); free(c->hostname); free(c->verify_name); @@ -725,8 +827,13 @@ static int scr_tls_f_sni(void *p, mbedtls_ssl_context *ssl, const unsigned char * random, session id, cipher suites, compression, then the extension list * — server_name (type 0) holds one host_name (type 0) entry. Answers 1 * with `out` set, 2 for a well-formed hello without SNI or anything - * malformed (the real handshake will judge it). */ -static int scr_tls_parse_ch_body(const unsigned char *b, size_t n, char *out, size_t out_cap) { + * malformed (the real handshake will judge it). *offers12 answers whether + * the hello offers TLS 1.2: true without a supported_versions extension + * (type 43), else whether its list holds 0x0303. */ +static int scr_tls_parse_ch_body(const unsigned char *b, size_t n, char *out, size_t out_cap, + bool *offers12) { + *offers12 = true; + int found = 2; size_t i = 34; /* version(2) + random(32) */ if (i + 1 > n) return 2; i += 1 + b[i]; /* session id: length byte + body */ @@ -744,24 +851,35 @@ static int scr_tls_parse_ch_body(const unsigned char *b, size_t n, char *out, si size_t el = ((size_t)b[i + 2] << 8) | b[i + 3]; i += 4; if (i + el > end) return 2; - if (et == 0) { + if (et == 0 && found == 2) { /* server_name: list length(2), then entry type(1) + length(2) + name */ - if (el < 5 || b[i + 2] != 0) return 2; - size_t nl = ((size_t)b[i + 3] << 8) | b[i + 4]; - if (nl == 0 || nl + 5 > el || nl + 1 > out_cap) return 2; - memcpy(out, b + i + 5, nl); - out[nl] = 0; - return 1; + if (el >= 5 && b[i + 2] == 0) { + size_t nl = ((size_t)b[i + 3] << 8) | b[i + 4]; + if (nl > 0 && nl + 5 <= el && nl + 1 <= out_cap) { + memcpy(out, b + i + 5, nl); + out[nl] = 0; + found = 1; + } + } + } else if (et == 43 && el >= 1) { + /* supported_versions: list length(1), then 2-byte versions */ + size_t vl = b[i] < el - 1 ? b[i] : el - 1; + *offers12 = false; + for (size_t v = 0; v + 1 < vl; v += 2) { + if (b[i + 1 + v] == 0x03 && b[i + 2 + v] == 0x03) *offers12 = true; + } } i += el; } - return 2; + return found; } /* Reassemble the handshake stream from the raw record flight and look for * the server_name extension. 0 = need more bytes; 1 = found (out set); * 2 = no SNI / not parseable (serve the default cert). */ -static int scr_tls_parse_sni(const unsigned char *buf, size_t len, char *out, size_t out_cap) { +static int scr_tls_parse_sni(const unsigned char *buf, size_t len, char *out, size_t out_cap, + bool *offers12) { + *offers12 = true; unsigned char *hs = NULL; size_t hs_len = 0, hs_cap = 0; size_t off = 0; @@ -785,7 +903,7 @@ static int scr_tls_parse_sni(const unsigned char *buf, size_t len, char *out, si size_t chlen = ((size_t)hs[1] << 16) | ((size_t)hs[2] << 8) | hs[3]; if (chlen > 131072) { free(hs); return 2; } if (hs_len >= 4 + chlen) { - int r = scr_tls_parse_ch_body(hs + 4, chlen, out, out_cap); + int r = scr_tls_parse_ch_body(hs + 4, chlen, out, out_cap, offers12); free(hs); return r; } @@ -811,8 +929,19 @@ static int scr_tls_sni_collect(ScrTlsSock *t) { char servername[256]; for (;;) { if (t->hello_len > 0) { - int r = scr_tls_parse_sni(t->hello, t->hello_len, servername, sizeof servername); - if (r == 2 || t->hello_len > 65536) { + bool offers12; + int r = scr_tls_parse_sni(t->hello, t->hello_len, servername, sizeof servername, &offers12); + if (r != 0 && t->srv->max_tls12 && !offers12) { + /* A TLS 1.2 server and a hello offering only later versions: + * OpenSSL answers with a fatal protocol_version alert, which the + * client reports as ERR_SSL_TLSV1_ALERT_PROTOCOL_VERSION. */ + static const unsigned char alert[] = { 0x15, 0x03, 0x03, 0x00, 0x02, 0x02, 0x46 }; + int fd = scr_net_sock_fd(t->sock); + if (fd >= 0) (void)!write(fd, alert, sizeof alert); + scr_net_sock_transport_error(t->sock, NULL); + return -1; + } + if (r == 2 || t->hello_len > 65536 || (r == 1 && t->srv->sni_cb == NULL)) { /* no SNI (or unparseable): skip the callback, serve the default * pair — Node's SNICallback never fires without a server_name */ t->sni_state = SCR_SNI_DONE; @@ -899,6 +1028,12 @@ void scr_tls_sni_answer(ScrClosure *self, bool has_err, ScrSecureCtx *ctx /*move static int scr_tls_handshake(void *tctx) { ScrTlsSock *t = (ScrTlsSock *)tctx; + if (t->cli != NULL && t->cli->no_protocols) { + /* OpenSSL's reason without its per-process prefix (SEMANTICS.md + * divergence 55); scr_error_message_code gives Node's code. */ + scr_net_sock_transport_error(t->sock, "error:0A0000BF:SSL routines::no protocols available"); + return -1; + } if (t->sni_state == SCR_SNI_COLLECT) { t->in_handshake = true; int c = scr_tls_sni_collect(t); @@ -927,6 +1062,20 @@ static int scr_tls_handshake(void *tctx) { * at this level and let the closed hook classify it */ scr_net_sock_transport_error(t->sock, NULL); return -1; + } else if (r == MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE && + t->ssl.MBEDTLS_PRIVATE(in_msg) != NULL && + t->ssl.MBEDTLS_PRIVATE(in_msg)[1] == MBEDTLS_SSL_ALERT_MSG_NO_APPLICATION_PROTOCOL) { + /* The server shares none of the ALPNProtocols: OpenSSL's reason, which + * Node reports as ERR_SSL_TLSV1_ALERT_NO_APPLICATION_PROTOCOL + * (scr_error_message_code), without its per-process prefix and source + * location (SEMANTICS.md divergence 55). */ + snprintf(msg, sizeof msg, + "error:0A000460:SSL routines:ssl3_read_bytes:tlsv1 alert no application protocol"); + } else if (r == MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE && + t->ssl.MBEDTLS_PRIVATE(in_msg) != NULL && + t->ssl.MBEDTLS_PRIVATE(in_msg)[1] == MBEDTLS_SSL_ALERT_MSG_PROTOCOL_VERSION) { + /* The server accepts none of the offered versions. */ + snprintf(msg, sizeof msg, "error:0A00042E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version"); } else { /* non-verify handshake failures: honest but not Node's exact openssl * string (SEMANTICS.md divergence 55) */ @@ -1012,19 +1161,44 @@ static void scr_tls_srv_on_conn(void *ctx, ScrNetSocket *sock) { if (mbedtls_ssl_setup(&t->ssl, &srv->conf) != 0) scr_tls_oom(); t->sock = sock; t->srv = scr_tls_srv_retain(srv); - if (srv->sni_cb != NULL) t->sni_state = SCR_SNI_COLLECT; /* pre-parse the hello */ + if (srv->sni_cb != NULL || srv->max_tls12) t->sni_state = SCR_SNI_COLLECT; /* pre-parse the hello */ mbedtls_ssl_set_bio(&t->ssl, t, scr_tls_bio_send, scr_tls_bio_recv, NULL); scr_net_sock_set_transport(sock, &SCR_TLS_OPS, t); } +static ScrNetServer *scr_tls_create_server_alpn(const char *cert, size_t cert_len, + const char *key, size_t key_len, + ScrClosure *handler /*moves, nullable*/, + ScrNetConnFn fn, char **alpn /*moves, nullable*/, + int min_version, int max_version) { + ScrTlsSrv *srv = scr_tls_srv_new(cert, cert_len, key, key_len, false); + srv->max_tls12 = max_version == 12; + if (!scr_tls_conf_versions(&srv->conf, min_version, max_version)) { + /* Node's server then fails every handshake; nothing here models + * that, so it stays a refusal. */ + scr_tls_srv_release(srv); + scr_closure_release(handler); + scr_tls_alpn_free(alpn); + scr_tls_opt_fence("tls.createServer", "maxVersion", + "a version range with neither TLSv1.2 nor TLSv1.3 has no lowering"); + return NULL; + } + if (alpn != NULL) { + /* mbedTLS picks the first of these the client offers (Node's server + * preference) and sends no_application_protocol when it offers none. */ + srv->alpn = alpn; + mbedtls_ssl_conf_alpn_protocols(&srv->conf, (const char **)srv->alpn); + } + ScrNetServer *s = scr_net_create_server(handler, fn); + scr_net_server_set_native_conn(s, &scr_tls_srv_on_conn, srv, &scr_tls_srv_release_v); + scr_net_server_defer_connections(s); /* the handler is 'secureConnection' */ + return s; +} + ScrNetServer *scr_tls_create_server(const char *cert, size_t cert_len, const char *key, size_t key_len, ScrClosure *handler /*moves, nullable*/, ScrNetConnFn fn) { - ScrTlsSrv *srv = scr_tls_srv_new(cert, cert_len, key, key_len, false); - ScrNetServer *s = scr_net_create_server(handler, fn); - scr_net_server_set_native_conn(s, &scr_tls_srv_on_conn, srv, &scr_tls_srv_release_v); - scr_net_server_defer_connections(s); /* the handler is 'secureConnection' */ - return s; + return scr_tls_create_server_alpn(cert, cert_len, key, key_len, handler, fn, NULL, 12, 13); } ScrNetServer *scr_https_create_server(const char *cert, size_t cert_len, const char *key, @@ -1730,8 +1904,10 @@ static bool scr_tls_opts_validate(const ScrDyn *opts) { * dyn options record; false = exception pending (partial results released). * Exported runtime-internally: scr_http2.c's createSecureServerDyn rides * the same walk (scr_http2.c always links alongside this unit). */ -bool scr_tls_srv_opts_walk(const ScrDyn *opts, const char *api, ScrBytes **cert_out, - ScrBytes **key_out) { +/* `server_ok`: tls.createServer reads ALPNProtocols, minVersion and + * maxVersion itself; the walk's other callers keep their fences. */ +static bool scr_tls_srv_opts_walk_ex(const ScrDyn *opts, const char *api, ScrBytes **cert_out, + ScrBytes **key_out, bool server_ok) { *cert_out = NULL; *key_out = NULL; if (opts == NULL || opts->kind != SCR_DYN_OBJ) { @@ -1770,6 +1946,9 @@ bool scr_tls_srv_opts_walk(const ScrDyn *opts, const char *api, ScrBytes **cert_ if (strcmp(e->key, "rejectUnauthorized") == 0) { continue; /* meaningful only with requestCert, which fences */ } + if (server_ok && (strcmp(e->key, "ALPNProtocols") == 0 || strcmp(e->key, "minVersion") == 0 || + strcmp(e->key, "maxVersion") == 0)) + continue; if (strcmp(e->key, "SNICallback") == 0) { scr_tls_opt_fence(api, "SNICallback", "SNICallback lowers only on http2.createSecureServer({ allowHTTP1: true, ... }) with a literal options object"); @@ -1803,6 +1982,11 @@ bool scr_tls_srv_opts_walk(const ScrDyn *opts, const char *api, ScrBytes **cert_ return ok; } +bool scr_tls_srv_opts_walk(const ScrDyn *opts, const char *api, ScrBytes **cert_out, + ScrBytes **key_out) { + return scr_tls_srv_opts_walk_ex(opts, api, cert_out, key_out, false); +} + /* createSecureContext over a RUNTIME options record: Node's typed * validations first (scr_tls_opts_validate), then the pem walk — a bag * that validates AND carries both cert and key builds the real context; @@ -1845,13 +2029,29 @@ void scr_tls_ca_certs_chk(const ScrDyn *type, const ScrStr *fence) { ScrNetServer *scr_tls_create_server_dyn(const ScrDyn *opts /*borrowed*/, ScrClosure *handler /*moves, nullable*/, ScrNetConnFn fn) { - ScrBytes *cert, *key; - if (!scr_tls_srv_opts_walk(opts, "tls.createServer", &cert, &key)) { + int min_version = 12, max_version = 13; + if (!scr_tls_versions_dyn(opts, &min_version, &max_version)) { scr_closure_release(handler); return NULL; } - ScrNetServer *s = scr_tls_create_server((const char *)cert->data, cert->len, - (const char *)key->data, key->len, handler, fn); + ScrBytes *cert, *key; + if (!scr_tls_srv_opts_walk_ex(opts, "tls.createServer", &cert, &key, true)) { + scr_closure_release(handler); + return NULL; + } + /* The Server constructor converts ALPNProtocols after the secure + * context validates (Node's order). */ + char **alpn = NULL; + const ScrDyn *protocols = scr_dyn_obj_get(opts, "ALPNProtocols", 13); + if (protocols != NULL && !scr_tls_alpn_dyn(protocols, &alpn)) { + scr_bytes_release(cert); + scr_bytes_release(key); + scr_closure_release(handler); + return NULL; + } + ScrNetServer *s = scr_tls_create_server_alpn((const char *)cert->data, cert->len, + (const char *)key->data, key->len, handler, fn, + alpn, min_version, max_version); scr_bytes_release(cert); scr_bytes_release(key); return s; @@ -1896,20 +2096,157 @@ ScrNetServer *scr_https_create_server_dyn(const ScrDyn *opts /*borrowed*/, /* ── tls.connect (the TLS client socket) ──────────────────────────────── */ +/* Node's inspect of a Buffer inside ERR_INVALID_ARG_VALUE's message: + * "" with at most 50 bytes, the whole rendering cut at + * 128 characters with "...". */ +static void scr_tls_alpn_value_fail(const char *reason, const uint8_t *data, size_t len) { + char inspected[400]; + size_t n = (size_t)snprintf(inspected, sizeof inspected, " shown) n += (size_t)snprintf(inspected + n, sizeof inspected - n, " ... %zu more byte%s", len - shown, len - shown == 1 ? "" : "s"); + snprintf(inspected + n, sizeof inspected - n, ">"); + if (strlen(inspected) > 128) strcpy(inspected + 128, "..."); + char msg[600]; + int m = snprintf(msg, sizeof msg, "The argument 'ALPNProtocols' %s. Received %s", reason, inspected); + scr_throw_error_msg_code(SCR_ERR_TYPE, msg, (size_t)m, "ERR_INVALID_ARG_VALUE"); +} + +/* The wire list (...) as mbedTLS's NULL-terminated names, after + * Node's validateALPNBuffer. An empty list offers nothing (*out stays + * NULL). False with the exception pending. */ +static bool scr_tls_alpn_from_wire(const uint8_t *data, size_t len, char ***out) { + size_t count = 0; + for (size_t at = 0; at < len;) { + size_t n = data[at]; + if (n == 0) { + scr_tls_alpn_value_fail("must not contain zero-length protocol", data, len); + return false; + } + if (at + 1 + n > len) { + scr_tls_alpn_value_fail("contains truncated protocol", data, len); + return false; + } + at += 1 + n; + count++; + } + if (count == 0) return true; + char **names = calloc(count + 1, sizeof *names); + if (!names) scr_tls_oom(); + size_t k = 0; + for (size_t at = 0; at < len; at += 1 + data[at]) { + names[k] = malloc((size_t)data[at] + 1); + if (!names[k]) scr_tls_oom(); + memcpy(names[k], data + at + 1, data[at]); + names[k][data[at]] = '\0'; + k++; + } + *out = names; + return true; +} + +/* tls.connect's ALPNProtocols (Node's convertALPNProtocols, then + * setALPNProtocols): an array of non-empty strings of at most 255 bytes, a + * Uint8Array or other ArrayBufferView in the wire format, or a falsy value + * (no ALPN). Any other truthy value is the binding's TypeError. False with + * the exception pending. */ +static bool scr_tls_alpn_dyn(const ScrDyn *v, char ***out) { + *out = NULL; + ScrDyn *own = NULL; + if (v->kind == SCR_DYN_TYPED_REF) { + own = scr_dyn_typed_ref_materialize(v); + if (own == NULL) return false; + v = own; + } + bool ok = true; + if (v->kind == SCR_DYN_ARR) { + /* convertProtocols: Buffer.byteLength of every entry first (strings, + * or the ArrayBuffer and view entries byteLength also measures), then + * each length's checks, then buff.write, which takes strings only. */ + size_t count = v->v.arr.len; + size_t total = 0; + for (size_t i = 0; ok && i < count; i++) { + const ScrDyn *e = v->v.arr.items[i]; + size_t n; + if (e->kind == SCR_DYN_STR) { + n = e->v.str->len; + } else if (e->kind == SCR_DYN_BYTES || scr_buffer_storage_is(e)) { + ScrBytes *view = scr_bytes_buffer_source(e); + if (view == NULL) { + ok = false; + break; + } + n = view->len; + scr_bytes_release(view); + } else { + scr_dyn_arg_type_fail("string", "of type string or an instance of Buffer or ArrayBuffer", e); + ok = false; + break; + } + if (n == 0) { + char name[32]; + snprintf(name, sizeof name, "protocols[%zu]", i); + scr_dyn_arg_value_fail(name, "must be a non-empty string", e); + ok = false; + break; + } + if (n > 255) { + char msg[160]; + int m = snprintf(msg, sizeof msg, + "The byte length of the protocol at index %zu exceeds the maximum length. It must be <= 255. Received %zu", + i, n); + scr_throw_error_msg_code(SCR_ERR_RANGE, msg, (size_t)m, "ERR_OUT_OF_RANGE"); + ok = false; + break; + } + total += 1 + n; + } + uint8_t *wire = ok ? malloc(total + 1) : NULL; + if (ok && !wire) scr_tls_oom(); + size_t at = 0; + for (size_t i = 0; ok && i < count; i++) { + const ScrDyn *e = v->v.arr.items[i]; + if (e->kind != SCR_DYN_STR) { + static const char msg[] = "argument must be a string"; + scr_throw_error_msg_code(SCR_ERR_TYPE, msg, sizeof msg - 1, "ERR_INVALID_ARG_TYPE"); + ok = false; + break; + } + wire[at++] = (uint8_t)e->v.str->len; + memcpy(wire + at, e->v.str->data, e->v.str->len); + at += e->v.str->len; + } + if (ok) ok = scr_tls_alpn_from_wire(wire, at, out); + free(wire); + } else if (v->kind == SCR_DYN_BYTES) { + ScrBytes *view = scr_bytes_buffer_source(v); + if (view == NULL) ok = false; + else { + ok = scr_tls_alpn_from_wire(view->data, view->len, out); + scr_bytes_release(view); + } + } else if (scr_tls_dyn_truthy(v)) { + static const char msg[] = "Must give a Buffer as first argument"; + scr_throw_error_msg(SCR_ERR_TYPE, msg, sizeof msg - 1); + ok = false; + } + scr_dyn_release(own); + return ok; +} + /* tls.connect's documented keys outside the implemented set: present * throws (the server walk's stance). */ static const char *const SCR_TLS_CONNECT_FENCED_OPTIONS[] = { - "ALPNProtocols", "allowHalfOpen", "autoSelectFamily", + "allowHalfOpen", "autoSelectFamily", "autoSelectFamilyAttemptTimeout", "cert", "certificateCompression", "checkServerIdentity", "ciphers", "clientCertEngine", "crl", "dhparam", "ecdhCurve", "enableTrace", "family", "fd", "hints", "highWaterMark", "honorCipherOrder", "keepAlive", "keepAliveInitialDelay", "key", - "localAddress", "localPort", "lookup", "maxVersion", "minDHSize", - "minVersion", "noDelay", "onread", "passphrase", "path", "pfx", + "localAddress", "localPort", "lookup", "minDHSize", "noDelay", "onread", "passphrase", "path", "pfx", "privateKeyEngine", "privateKeyIdentifier", "pskCallback", "readable", "secureContext", "secureOptions", "secureProtocol", "session", "sessionIdContext", "sigalgs", "signal", "socket", "ticketKeys", - "timeout", "writable", NULL, + "writable", NULL, }; ScrNetSocket *scr_tls_connect_dyn(double port, ScrStr *host /*borrowed, nullable*/, @@ -1920,12 +2257,19 @@ ScrNetSocket *scr_tls_connect_dyn(double port, ScrStr *host /*borrowed, nullable ScrBytes *ca = NULL; const ScrStr *servername = NULL; const ScrStr *opt_host = NULL; + char **alpn = NULL; + const ScrDyn *alpn_value = NULL; + const ScrDyn *timeout_value = NULL; + int min_version = 12, max_version = 13; bool ok = true; if (opts != NULL && opts->kind != SCR_DYN_UNDEF && opts->kind != SCR_DYN_NULL) { if (opts->kind != SCR_DYN_OBJ) { scr_dyn_arg_type_fail("options", "of type object", opts); ok = false; } + /* createSecureContext's version checks come before its other options + * (Node's SecureContext constructor). */ + if (ok && !scr_tls_versions_dyn(opts, &min_version, &max_version)) ok = false; for (size_t i = 0; ok && i < opts->v.obj.len; i++) { const ScrDynEntry *e = &opts->v.obj.entries[i]; const ScrDyn *v = e->value; @@ -1974,6 +2318,16 @@ ScrNetSocket *scr_tls_connect_dyn(double port, ScrStr *host /*borrowed, nullable if (ca == NULL) ok = false; continue; } + if (strcmp(e->key, "ALPNProtocols") == 0) { + alpn_value = v; /* converted after the context options, as TLSSocket does */ + continue; + } + if (strcmp(e->key, "minVersion") == 0 || strcmp(e->key, "maxVersion") == 0) continue; + if (strcmp(e->key, "timeout") == 0) { + /* `if (options.timeout) tlssock.setTimeout(options.timeout)` */ + timeout_value = scr_tls_dyn_truthy(v) ? v : NULL; + continue; + } if (strcmp(e->key, "servername") == 0) { if (v->kind == SCR_DYN_STR) servername = v->v.str; else { @@ -1985,7 +2339,7 @@ ScrNetSocket *scr_tls_connect_dyn(double port, ScrStr *host /*borrowed, nullable for (size_t k = 0; SCR_TLS_CONNECT_FENCED_OPTIONS[k] != NULL; k++) { if (strcmp(e->key, SCR_TLS_CONNECT_FENCED_OPTIONS[k]) == 0) { scr_tls_opt_fence("tls.connect", e->key, - "port/host/rejectUnauthorized/ca/servername are the honestly-implemented members of a runtime options record"); + "port/host/rejectUnauthorized/ca/servername/ALPNProtocols/minVersion/maxVersion/timeout are the implemented members of a runtime options record"); ok = false; break; } @@ -1993,6 +2347,26 @@ ScrNetSocket *scr_tls_connect_dyn(double port, ScrStr *host /*borrowed, nullable /* anything else: an undocumented key, dropped like Node drops it */ } } + if (ok && alpn_value != NULL && !scr_tls_alpn_dyn(alpn_value, &alpn)) ok = false; + /* socket.setTimeout(msecs) runs after the TLSSocket exists and before it + * connects: Node's msecs checks, then the inactivity timer below. */ + double timeout_ms = 0; + if (ok && timeout_value != NULL) { + if (timeout_value->kind != SCR_DYN_NUM) { + scr_dyn_arg_type_fail("msecs", "of type number", timeout_value); + ok = false; + } else if (!isfinite(timeout_value->v.num) || timeout_value->v.num < 0) { + char recv[48], msg[160]; + scr_num_received(timeout_value->v.num, recv); + int m = snprintf(msg, sizeof msg, + "The value of \"msecs\" is out of range. It must be a non-negative finite number. Received %s", + recv); + scr_throw_error_msg_code(SCR_ERR_RANGE, msg, (size_t)m, "ERR_OUT_OF_RANGE"); + ok = false; + } else { + timeout_ms = timeout_value->v.num; + } + } if (ok && port < 0) { scr_dyn_arg_type_fail("options.port", "of type number", scr_dyn_undefined()); ok = false; @@ -2000,6 +2374,7 @@ ScrNetSocket *scr_tls_connect_dyn(double port, ScrStr *host /*borrowed, nullable ScrStr *dial_host = (ScrStr *)(host != NULL && host->len > 0 ? host : opt_host); if (ok && !scr_net_host_chk(dial_host)) ok = false; if (!ok) { + scr_tls_alpn_free(alpn); scr_bytes_release(ca); scr_closure_release(cb); return NULL; @@ -2022,7 +2397,13 @@ ScrNetSocket *scr_tls_connect_dyn(double port, ScrStr *host /*borrowed, nullable ca != NULL ? (const char *)ca->data : NULL, ca != NULL ? ca->len : 0, !reject); if (servername != NULL) scr_tls_cli_set_servername(cli, servername); + if (alpn != NULL) { + cli->alpn = alpn; /* moves */ + mbedtls_ssl_conf_alpn_protocols(&cli->conf, (const char **)cli->alpn); + } + cli->no_protocols = !scr_tls_conf_versions(&cli->conf, min_version, max_version); scr_tls_cli_wrap(s, cli); + if (timeout_ms > 0) scr_net_sock_set_timeout(s, timeout_ms); scr_str_release(localhost_fallback); scr_bytes_release(ca); return s; diff --git a/tests/corpus/6721-tls-alpn-protocols.ts b/tests/corpus/6721-tls-alpn-protocols.ts new file mode 100644 index 000000000..f68fb2840 --- /dev/null +++ b/tests/corpus/6721-tls-alpn-protocols.ts @@ -0,0 +1,75 @@ +// tls.connect and tls.createServer with ALPNProtocols: the server answers +// with the first of its own protocols the client offers, both sockets +// report it as alpnProtocol, and a client that offers none of the +// server's fails with ERR_SSL_TLSV1_ALERT_NO_APPLICATION_PROTOCOL. An +// array of strings, a Buffer or a Uint8Array in the wire format, an empty +// list and a falsy value all work as in Node, and invalid values throw +// Node's errors from tls.connect. +import * as tls from "node:tls"; +import { readFileSync } from "node:fs"; + +const dir = "tests/fixtures/server/certs/"; +const ca = readFileSync(dir + "ca.pem"); +const server = tls.createServer( + { + cert: readFileSync(dir + "localhost.pem"), + key: readFileSync(dir + "localhost-key.pem"), + ALPNProtocols: ["h2", "x/1"], + }, + (socket) => { + socket.end(`server:${String(socket.alpnProtocol)}`); + }, +); + +function attempt(port: number, label: string, protocols: unknown): Promise { + return new Promise((resolve) => { + const socket = tls.connect({ host: "localhost", port, ca, ALPNProtocols: protocols as any }, () => { + console.log(label, "client:", JSON.stringify(socket.alpnProtocol)); + }); + let data = ""; + socket.on("data", (chunk: Buffer) => (data += chunk.toString())); + socket.on("end", () => { + console.log(label, data); + resolve(); + }); + socket.on("error", (err) => { + console.log(label, "error:", (err as NodeJS.ErrnoException).code); + resolve(); + }); + }); +} + +function invalid(label: string, protocols: unknown): void { + try { + const socket = tls.connect({ host: "127.0.0.1", port: 1, ALPNProtocols: protocols as any }); + socket.on("error", () => {}); + socket.destroy(); + console.log(label, "accepted"); + } catch (e) { + const err = e as NodeJS.ErrnoException; + console.log(label, err.name, err.code, err.message); + } +} + +server.listen(0, "127.0.0.1", async () => { + const port = (server.address() as { port: number }).port; + await attempt(port, "strings", ["x/1", "h2"]); + await attempt(port, "only-second", ["zz", "x/1"]); + await attempt(port, "buffer", Buffer.from([2, 0x68, 0x32])); + await attempt(port, "uint8array", new Uint8Array([3, 0x78, 0x2f, 0x31])); + await attempt(port, "absent", undefined); + await attempt(port, "empty", []); + await attempt(port, "falsy", 0); + await attempt(port, "no-match", ["zz"]); + server.close(); + + invalid("string", "h2"); + invalid("object", {}); + invalid("buffer-element", [Buffer.from("h2")]); + invalid("number-element", [1]); + invalid("empty-element", ["h2", ""]); + invalid("too-long", ["a".repeat(256)]); + invalid("zero-length", Buffer.from([0])); + invalid("truncated", Buffer.from([5, 0x68])); + invalid("long-zero", Buffer.alloc(60)); +}); diff --git a/tests/corpus/6722-tls-min-max-version.ts b/tests/corpus/6722-tls-min-max-version.ts new file mode 100644 index 000000000..07d01867d --- /dev/null +++ b/tests/corpus/6722-tls-min-max-version.ts @@ -0,0 +1,85 @@ +// tls.connect and tls.createServer with minVersion and maxVersion: the +// handshake picks the highest version both ranges allow and getProtocol() +// reports it. A range the server does not share fails with +// ERR_SSL_TLSV1_ALERT_PROTOCOL_VERSION, a client range with neither TLSv1.2 +// nor TLSv1.3 fails with ERR_SSL_NO_PROTOCOLS_AVAILABLE once the handshake +// starts, and an invalid version or a secureProtocol conflict throws Node's +// TypeError from tls.connect. +import * as tls from "node:tls"; +import { readFileSync } from "node:fs"; + +const dir = "tests/fixtures/server/certs/"; +const ca = readFileSync(dir + "ca.pem"); +const cert = readFileSync(dir + "localhost.pem"); +const key = readFileSync(dir + "localhost-key.pem"); + +const upTo12 = tls.createServer({ cert, key, maxVersion: "TLSv1.2" }, (s) => s.end(`server:${s.getProtocol()}`)); +const from13 = tls.createServer({ cert, key, minVersion: "TLSv1.3" }, (s) => s.end(`server:${s.getProtocol()}`)); +const any = tls.createServer({ cert, key }, (s) => s.end(`server:${s.getProtocol()}`)); + +function listen(server: tls.Server): Promise { + return new Promise((resolve) => + server.listen(0, "127.0.0.1", () => resolve((server.address() as { port: number }).port)), + ); +} + +interface Versions { + minVersion?: unknown; + maxVersion?: unknown; + secureProtocol?: string; +} + +function attempt(label: string, port: number, versions: Versions): Promise { + return new Promise((resolve) => { + let socket: tls.TLSSocket; + try { + const options = { + host: "localhost", + port, + ca, + minVersion: versions.minVersion as any, + maxVersion: versions.maxVersion as any, + secureProtocol: versions.secureProtocol, + }; + socket = tls.connect(options, () => console.log(label, "client:", socket.getProtocol())); + } catch (e) { + const err = e as NodeJS.ErrnoException; + console.log(label, err.name, err.code, err.message); + resolve(); + return; + } + let data = ""; + socket.on("data", (chunk: Buffer) => (data += chunk.toString())); + socket.on("end", () => { + console.log(label, data); + resolve(); + }); + socket.on("error", (err) => { + console.log(label, "error:", (err as NodeJS.ErrnoException).code); + resolve(); + }); + }); +} + +async function main(): Promise { + const p12 = await listen(upTo12); + const p13 = await listen(from13); + const pAny = await listen(any); + await attempt("defaults", pAny, {}); + await attempt("max-1.2", pAny, { maxVersion: "TLSv1.2" }); + await attempt("min-1.3", pAny, { minVersion: "TLSv1.3" }); + await attempt("min-1-against-1.2", p12, { minVersion: "TLSv1" }); + await attempt("min-1.3-against-1.2", p12, { minVersion: "TLSv1.3" }); + await attempt("max-1.2-against-1.3", p13, { maxVersion: "TLSv1.2" }); + await attempt("max-1.1", pAny, { maxVersion: "TLSv1.1" }); + await attempt("min-above-max", pAny, { minVersion: "TLSv1.3", maxVersion: "TLSv1.2" }); + await attempt("null-min", pAny, { minVersion: null }); + await attempt("invalid-min", pAny, { minVersion: "TLSv9" }); + await attempt("invalid-max", pAny, { maxVersion: 3 }); + await attempt("conflict", pAny, { minVersion: "TLSv1.2", secureProtocol: "TLSv1_2_method" }); + upTo12.close(); + from13.close(); + any.close(); +} + +void main(); diff --git a/tests/corpus/6723-tls-connect-timeout.ts b/tests/corpus/6723-tls-connect-timeout.ts new file mode 100644 index 000000000..68f40ec37 --- /dev/null +++ b/tests/corpus/6723-tls-connect-timeout.ts @@ -0,0 +1,51 @@ +// tls.connect's timeout option arms the socket's inactivity timeout +// before it connects, so a handshake the peer never answers emits +// 'timeout'. Node applies it only when truthy, so 0 and NaN arm nothing, +// and setTimeout's msecs checks throw from tls.connect for a value that is +// not a number, negative, or not finite. +import * as net from "node:net"; +import * as tls from "node:tls"; + +// The server reads (and ignores) what arrives, so each connection closes +// when its client is destroyed. +const silent = net.createServer((socket) => { + socket.resume(); +}); + +function attempt(port: number, label: string, timeout: unknown): Promise { + return new Promise((resolve) => { + let socket: tls.TLSSocket; + try { + socket = tls.connect({ host: "127.0.0.1", port, timeout: timeout as any }); + } catch (e) { + const err = e as NodeJS.ErrnoException; + console.log(label, err.name, err.code, err.message); + resolve(); + return; + } + const fallback = setTimeout(() => { + console.log(label, "no timeout event"); + socket.destroy(); + resolve(); + }, 300); + socket.on("timeout", () => { + console.log(label, "timeout"); + clearTimeout(fallback); + socket.destroy(); + resolve(); + }); + socket.on("error", (err) => console.log(label, "error", (err as NodeJS.ErrnoException).code)); + }); +} + +silent.listen(0, "127.0.0.1", async () => { + const port = (silent.address() as net.AddressInfo).port; + await attempt(port, "50ms", 50); + await attempt(port, "fractional", 30.7); + await attempt(port, "zero", 0); + await attempt(port, "nan", NaN); + await attempt(port, "negative", -1); + await attempt(port, "infinite", Infinity); + await attempt(port, "string", "x"); + silent.close(); +}); diff --git a/tests/harness/corpus-costs/6721-tls-alpn-protocols.ts.json b/tests/harness/corpus-costs/6721-tls-alpn-protocols.ts.json new file mode 100644 index 000000000..ab6ff1c99 --- /dev/null +++ b/tests/harness/corpus-costs/6721-tls-alpn-protocols.ts.json @@ -0,0 +1 @@ +{ "plain": 4300, "san": 60600 } diff --git a/tests/harness/corpus-costs/6722-tls-min-max-version.ts.json b/tests/harness/corpus-costs/6722-tls-min-max-version.ts.json new file mode 100644 index 000000000..c1b1d2d17 --- /dev/null +++ b/tests/harness/corpus-costs/6722-tls-min-max-version.ts.json @@ -0,0 +1 @@ +{ "plain": 3000, "san": 4700 } diff --git a/tests/harness/corpus-costs/6723-tls-connect-timeout.ts.json b/tests/harness/corpus-costs/6723-tls-connect-timeout.ts.json new file mode 100644 index 000000000..9ebfb8b9d --- /dev/null +++ b/tests/harness/corpus-costs/6723-tls-connect-timeout.ts.json @@ -0,0 +1 @@ +{ "plain": 6200, "san": 60300 } From e41858c7dfa521e387f1ddfef57632a117514540 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 08:58:51 -0500 Subject: [PATCH 15/51] Compile assignment and update targets wrapped in parentheses or non-null assertions TypeScript accepts parentheses and non-null assertions around the reference an assignment, a `++`/`--` update or a for-in/of head writes (`(x) = 1`, `h.sums[k]! += 10`), and both are erased at run time. The compiler now reads its source files with the AstFile writeTargets option: the writer's `left`, `operand` or `initializer` is the reference itself and the reference's `parent` is the writer, so lowering and every analysis of a binding's or member's writes see one shape. The reference keeps its own source span. Files read without the option, such as type nodes from the checker and the AST conformance tests, keep TypeScript's exact tree. Fixes #1625 Co-authored-by: ejc3 <1694850+ejc3@users.noreply.github.com> --- .../src/frontend/ts7/ast-node.test.ts | 34 ++++++ .../compiler/src/frontend/ts7/ast-node.ts | 106 +++++++++++++++++- packages/compiler/src/frontend/ts7/session.ts | 10 +- .../corpus/wrapped-assignment-targets.ts.json | 6 + tests/corpus/wrapped-assignment-targets.ts | 54 +++++++++ 5 files changed, 203 insertions(+), 7 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/wrapped-assignment-targets.ts.json create mode 100644 tests/corpus/wrapped-assignment-targets.ts diff --git a/packages/compiler/src/frontend/ts7/ast-node.test.ts b/packages/compiler/src/frontend/ts7/ast-node.test.ts index b9b8f0e7b..60bfaa005 100644 --- a/packages/compiler/src/frontend/ts7/ast-node.test.ts +++ b/packages/compiler/src/frontend/ts7/ast-node.test.ts @@ -101,6 +101,8 @@ const cases: Record = { '/** @typedef {Object} Options\n * @property {string} name\n * @property {number} count\n */\n/** @type {Options} */ const options = { name: "x", count: 2 };', "missing.ts": "const incomplete = ;\nfunction f( {\n", "empty.ts": "", + "targets.ts": + "let x: number | undefined = 1;\nconst a = [1];\n(x) = 2;\nx! = 3;\na[0]! += 1;\n(a[0])!++;\n--(x!);\nfor ((x) of a) {}\nconst y = (x!) + 1;\n", "lists.ts": "const trailing = [1, 2 /* last */, ];\nconst plain = [1, 2];\nf(1, 2,);\nexport abstract class Flags { protected static readonly value = 1; }\n", }; @@ -167,6 +169,38 @@ function ids( return nodes?.map((n) => (n as OracleNode).index); } +test("the writeTargets option answers write targets without their wrappers", () => { + const { bytes, file: plain } = decoded.get("targets.ts")!; + const file = new AstFile(bytes, undefined, { writeTargets: true }); + const statements = (root: AstNode) => + (root.childList("statements") as AstNodeList).nodes.map((statement) => + statement.kind === SyntaxKind.ExpressionStatement ? statement.expression : statement, + ); + const text = (node: AstNode | undefined) => node?.getText(); + const writes = statements(file.root); + const raw = statements(plain.root); + // Assignment left sides, update operands and a for-of head. + const targets = [writes[2]!.left, writes[3]!.left, writes[4]!.left, writes[5]!.operand]; + expect(targets.map(text)).toEqual(["x", "x", "a[0]", "a[0]"]); + expect(text(writes[6]!.operand)).toBe("x"); + expect(text(writes[7]!.initializer)).toBe("x"); + for (const [index, target] of targets.entries()) + expect(target!.parent, `target ${index}`).toBe(writes[index + 2]); + expect(writes[7]!.initializer!.parent).toBe(writes[7]); + // The reference keeps its own source span. + expect(writes[4]!.left!.getStart()).toBe(raw[4]!.left!.getStart()); + expect(writes[4]!.left!.end).toBe(raw[4]!.left!.end - 1); + // A wrapper outside a write target is kept. + const init = (writes[8]!.childNode("declarationList")!.childList("declarations") as AstNodeList) + .nodes[0]!.initializer!; + expect(text(init.left)).toBe("(x!)"); + expect(init.left!.kind).toBe(SyntaxKind.ParenthesizedExpression); + // Without the option, the file is TypeScript's exact tree. + expect(raw[3]!.left!.kind).toBe(SyntaxKind.NonNullExpression); + expect(raw[2]!.left!.kind).toBe(SyntaxKind.ParenthesizedExpression); + expect(raw[5]!.operand!.kind).toBe(SyntaxKind.NonNullExpression); +}); + test("node lists report a trailing comma and modifier lists their flags", () => { const { file } = decoded.get("lists.ts")!; const lists: [SyntaxKind, boolean][] = []; diff --git a/packages/compiler/src/frontend/ts7/ast-node.ts b/packages/compiler/src/frontend/ts7/ast-node.ts index 8eaa87c72..5fef740ae 100644 --- a/packages/compiler/src/frontend/ts7/ast-node.ts +++ b/packages/compiler/src/frontend/ts7/ast-node.ts @@ -60,6 +60,18 @@ export class AstModifierList extends AstNodeList * list. * Parent access is lazy, so resolving a checker handle near the bottom * of a deep tree never recursively constructs its ancestors. */ +export interface AstFileOptions { + /** Answer the reference an assignment, a `++`/`--` update or a for-in/of + * head writes without the parentheses and non-null assertions around it. + * TypeScript accepts both around a target (`(x) = 1`, `a[i]! += 1`) and + * both are erased at run time. With this option the writer's `left`, + * `operand` or `initializer` is the reference itself and the reference's + * `parent` is the writer, so every reader of a write target sees one + * shape. Positions stay the source's: the reference keeps its own span. + * Files read for other purposes keep TypeScript's exact tree. */ + writeTargets?: boolean; +} + export class AstFile { readonly wire: AstWireFile; readonly root: AstNode; @@ -73,10 +85,16 @@ export class AstFile { private lines: number[] | undefined; private childIndices: Uint32Array | undefined; + /** Whether nodes answer write targets without their wrappers (see + * AstFileOptions.writeTargets). */ + readonly writeTargets: boolean; + constructor( bytes: Uint8Array, private readonly materialized?: () => void, + options: AstFileOptions = {}, ) { + this.writeTargets = options.writeTargets === true; this.wire = new AstWireFile(bytes); this.nodes = new Array(this.wire.nodeCount); this.lists = new Array(this.wire.nodeCount); @@ -192,6 +210,73 @@ export class AstFile { /** Concrete native layout shared by every syntax kind. Kind-specific * properties are views over the wire; they never copy identity-bearing * nodes into structural records. */ +/* Write targets without their wrappers (AstFileOptions.writeTargets). */ + +function isTargetWrapper(node: AstNode): boolean { + return node.kind === AstKind.ParenthesizedExpression || node.kind === AstKind.NonNullExpression; +} + +function unwrapTarget(node: AstNode): AstNode { + let target = node; + while (isTargetWrapper(target)) { + const inner = target.childNode("expression"); + if (inner === undefined) break; + target = inner; + } + return target; +} + +/** Whether `field` of `node` is a write target: an assignment's left side, + * a `++`/`--` operand, or a for-in/of head that declares nothing. */ +function writesTarget(node: AstNode, field: "left" | "operand" | "initializer"): boolean { + switch (node.kind) { + case AstKind.BinaryExpression: { + if (field !== "left") return false; + const operator = node.childNode("operatorToken")?.kind; + return ( + operator !== undefined && + operator >= AstKind.FirstAssignment && + operator <= AstKind.LastAssignment + ); + } + case AstKind.PrefixUnaryExpression: + case AstKind.PostfixUnaryExpression: + return ( + field === "operand" && + (node.operator === AstKind.PlusPlusToken || node.operator === AstKind.MinusMinusToken) + ); + case AstKind.ForInStatement: + case AstKind.ForOfStatement: + return ( + field === "initializer" && + node.childNode("initializer")?.kind !== AstKind.VariableDeclarationList + ); + default: + return false; + } +} + +/** The parent of `node`, whose syntactic parent `wrapper` is a parenthesis + * or a non-null assertion: the writer when the wrappers around `node` are + * the whole target of an assignment, update or for-in/of head, and + * `wrapper` otherwise. */ +function writerOf(node: AstNode, wrapper: AstNode): AstNode { + let outer = wrapper; + for (;;) { + const up = outer.parent; + if (up === undefined) return wrapper; + if (isTargetWrapper(up)) { + outer = up; + continue; + } + for (const field of ["left", "operand", "initializer"] as const) { + if (writesTarget(up, field) && up.childNode(field) === outer && unwrapTarget(outer) === node) + return up; + } + return wrapper; + } +} + export class AstNode { readonly kind: SyntaxKind; readonly pos: number; @@ -221,7 +306,11 @@ export class AstNode { get parent(): AstNode | undefined { if (!this.parentResolved) { const index = this.file.wire.semanticParent(this.index); - this.parentCache = index === 0 || index === this.index ? undefined : this.file.node(index); + const parent = index === 0 || index === this.index ? undefined : this.file.node(index); + this.parentCache = + parent !== undefined && this.file.writeTargets && isTargetWrapper(parent) + ? writerOf(this, parent) + : parent; this.parentResolved = true; } return this.parentCache; @@ -612,7 +701,10 @@ export class AstNode { return this.childNode("indexType"); } get initializer(): AstNode | undefined { - return this.childNode("initializer"); + const initializer = this.childNode("initializer"); + return initializer !== undefined && this.file.writeTargets && writesTarget(this, "initializer") + ? unwrapTarget(initializer) + : initializer; } get jsdocPropertyTags(): AstNode | undefined { return this.childNode("jsdocPropertyTags"); @@ -621,7 +713,10 @@ export class AstNode { return this.childNode("label"); } get left(): AstNode | undefined { - return this.childNode("left"); + const left = this.childNode("left"); + return left !== undefined && this.file.writeTargets && writesTarget(this, "left") + ? unwrapTarget(left) + : left; } get literal(): AstNode | undefined { return this.childNode("literal"); @@ -663,7 +758,10 @@ export class AstNode { return this.childNode("openingFragment"); } get operand(): AstNode | undefined { - return this.childNode("operand"); + const operand = this.childNode("operand"); + return operand !== undefined && this.file.writeTargets && writesTarget(this, "operand") + ? unwrapTarget(operand) + : operand; } get operatorToken(): AstNode | undefined { return this.childNode("operatorToken"); diff --git a/packages/compiler/src/frontend/ts7/session.ts b/packages/compiler/src/frontend/ts7/session.ts index c58588f6d..d3f2ae9fe 100644 --- a/packages/compiler/src/frontend/ts7/session.ts +++ b/packages/compiler/src/frontend/ts7/session.ts @@ -298,9 +298,13 @@ export class Ts7SessionProgram { if (source === undefined) { const bytes = this.session.binary("getSourceFile", JSON.stringify(this.query(file))); if (bytes.length !== 0) { - const ast = new AstFile(bytes, () => { - this.session.timing.materialized(); - }); + const ast = new AstFile( + bytes, + () => { + this.session.timing.materialized(); + }, + { writeTargets: true }, + ); this.session.timing.fetched(Math.max(0, ast.wire.nodeCount - 2)); source = this.session.cache.set(path, ast.sourceFile, this.snapshot.id, this.project.id); } diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/wrapped-assignment-targets.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/wrapped-assignment-targets.ts.json new file mode 100644 index 000000000..6a4a8b936 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/wrapped-assignment-targets.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/wrapped-assignment-targets.ts" + ], + "diags": [] +} diff --git a/tests/corpus/wrapped-assignment-targets.ts b/tests/corpus/wrapped-assignment-targets.ts new file mode 100644 index 000000000..203a29429 --- /dev/null +++ b/tests/corpus/wrapped-assignment-targets.ts @@ -0,0 +1,54 @@ +// Assignment, update and for-of targets wrapped in parentheses or non-null +// assertions write the reference inside them, as in JavaScript, and the +// writes count for every analysis of the binding or member they write. +const h = { sums: [1, 2, 3] }; +const k = 1; +h.sums[k]! += 10; +console.log(h.sums.join(",")); +let x = 1; +(x) = 2; +console.log(x); +let y: number | undefined = 1; +y! = 5; +console.log(y); +const a = [1, 2]; +(a[0]) += 1; +a[1]! = 7; +a[0]!++; +++(a[1]!); +console.log(a.join(",")); +const o: { n?: number } = { n: 1 }; +o.n! = 4; +(o.n)! *= 2; +console.log(o.n); +let counter = 0; +for ((counter) of [3, 4]) {} +console.log(counter); +const m = new Map([["a", [1]]]); +m.get("a")![0]! += 4; +console.log(m.get("a")!.join(",")); +const r = (a[0]! += 2); +console.log(r, a[0]); +let limit = 3; +function bump(): void { + (limit) = 9; +} +bump(); +console.log(limit); +class Counter { + count: number | undefined = 0; + readonly fixed = 1; + inc(): void { + this.count! += 1; + (this.count)!++; + } +} +const c = new Counter(); +c.inc(); +console.log(c.count, c.fixed); +let flag = false; +const set = (): void => { + flag! = true; +}; +set(); +console.log(flag); From a7faf915156ec2c0988ed94072637aed10b492c1 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 08:59:12 -0500 Subject: [PATCH 16/51] Accept defaulted callbacks in literals under an erased generic member A generic arrow or method held by an object literal is also lowered with its type parameters erased, for the literal's own value. In that lowering the object it returns builds as a checked-dynamic value, so a callback member with an optional or defaulted parameter, such as `decode(value, at = [])`, met a slot type that maps to a dynamic value. The completed-signature check refused it with SC1090, although the same code in a standalone generic function compiled. A slot whose type maps to a dynamic value now takes the function the same way an 'any' slot does: the value boxes into a checked-dynamic function, which completes omitted arguments with undefined. Fixes #1373 Co-authored-by: blurname <50226652+blurname@users.noreply.github.com> --- .../src/frontend/lowering/call-signatures.ts | 6 ++-- ...-generic-member-defaulted-callback.ts.json | 6 ++++ .../6907-generic-member-defaulted-callback.ts | 33 +++++++++++++++++++ 3 files changed, 43 insertions(+), 2 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6907-generic-member-defaulted-callback.ts.json create mode 100644 tests/corpus/6907-generic-member-defaulted-callback.ts diff --git a/packages/compiler/src/frontend/lowering/call-signatures.ts b/packages/compiler/src/frontend/lowering/call-signatures.ts index e5ce784be..4e1ece0e9 100644 --- a/packages/compiler/src/frontend/lowering/call-signatures.ts +++ b/packages/compiler/src/frontend/lowering/call-signatures.ts @@ -676,12 +676,14 @@ export function requireExactArityValue( : null; if (mapped && typeEquals(mapped, funcType)) return; // An 'any' slot (`C.prototype.m = function (a, b) {…}` through an - // untyped receiver) boxes the value into a checked-dynamic function: + // untyped receiver), or a slot whose type maps to a checked-dynamic + // value (a member of a literal lowered with a generic function's type + // parameters erased), boxes the value into a checked-dynamic function: // the box completes omitted arguments with undefined and checks every // argument into the completed ABI, so any boxable signature passes. if ( target !== undefined && - (target.flags & ts.TypeFlags.Any) !== 0 && + ((target.flags & ts.TypeFlags.Any) !== 0 || mapped?.kind === "dyn") && canBoxFuncIntoDyn( funcType, (id) => lowerer.shapes.get(id), diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6907-generic-member-defaulted-callback.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6907-generic-member-defaulted-callback.ts.json new file mode 100644 index 000000000..bf034340e --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6907-generic-member-defaulted-callback.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6907-generic-member-defaulted-callback.ts" + ], + "diags": [] +} diff --git a/tests/corpus/6907-generic-member-defaulted-callback.ts b/tests/corpus/6907-generic-member-defaulted-callback.ts new file mode 100644 index 000000000..d9ca305a6 --- /dev/null +++ b/tests/corpus/6907-generic-member-defaulted-callback.ts @@ -0,0 +1,33 @@ +// A generic arrow or method held by an object literal returns an object +// whose callback has a defaulted parameter. The callback is typed by the +// declared result (`at?: readonly string[]`), and calls that omit the +// argument use the default, as in Node. +type Codec = { decode: (value: unknown, at?: readonly string[]) => T }; +function describe(codec: C): C { + return codec; +} + +const t = { + literal: (initial: T): Codec => + describe({ + decode: (_value: unknown, at: readonly string[] = []): T => { + console.log("at", at.length === 0 ? "(default)" : at.join(".")); + return initial; + }, + }), + pair(a: A, b: B): Codec<[A, B]> { + return { decode: (_value: unknown, at: readonly string[] = ["root"]) => (console.log(at.join(".")), [a, b]) }; + }, +}; + +console.log(t.literal(42).decode(null)); +console.log(t.literal("text").decode(null, ["a", "b"])); +const flag = t.literal(true); +console.log(flag.decode(undefined), flag.decode(1, [])); +console.log(JSON.stringify(t.pair(1, "x").decode(null))); +console.log(JSON.stringify(t.pair(false, [2]).decode(null, ["p"]))); + +function literal(initial: T): Codec { + return describe({ decode: (_value: unknown, _at: readonly string[] = []): T => initial }); +} +console.log(literal(7).decode(null)); From 482c2c01781561c4bfd40757b6049edfb415738e Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 09:04:23 -0500 Subject: [PATCH 17/51] Return the pages faulted in past a parked arena's frontiers When a script thread parks its allocator arena, each class slice still holds the pages past its bump pointer that were faulted in but never carved: the rest of the huge page the bump pointer is in, and the pages the pre-fault helper faulted in ahead of it. The same applies to region chunks when the thread exits. Until another thread claims the arena, nothing carves there, so a program that continues on threads that never claim it (parse threads ahead of a check on the main thread) kept them. - scr_sa_park_arena returns those pages with MADV_DONTNEED. The next thread to claim the arena faults them in again as it carves, and they read as zero, as fresh pages do. - scr_rgn_thread_exit returns each chunk's pages past its bump on worker threads; nothing carves in an exited thread's regions. - scr_mem_release_ahead drops pre-fault requests that overlap the range and waits for one in progress before returning it, so the helper does not fault the pages in again. - scr_alloc.test.c checks that a worker that shares and carves past the huge-page switch leaves no resident pages past its frontier, keeps its blocks, and that the next thread carves on from the frontier, zeroed. --- packages/runtime/src/scr_alloc.c | 71 ++++++++++++++++++++++++++- packages/runtime/src/scr_alloc.test.c | 63 ++++++++++++++++++++++++ packages/runtime/src/scr_rc.h | 4 ++ packages/runtime/src/scr_region.c | 12 +++++ 4 files changed, 149 insertions(+), 1 deletion(-) diff --git a/packages/runtime/src/scr_alloc.c b/packages/runtime/src/scr_alloc.c index 9f8f4c6c1..8cf14fb7f 100644 --- a/packages/runtime/src/scr_alloc.c +++ b/packages/runtime/src/scr_alloc.c @@ -226,6 +226,11 @@ static pthread_cond_t scr_sa_pf_cond = PTHREAD_COND_INITIALIZER; static ScrSaPfRange scr_sa_pf_ring[SCR_SA_PF_RING]; static unsigned scr_sa_pf_head, scr_sa_pf_tail; /* under the lock */ static bool scr_sa_pf_started, scr_sa_pf_waiting; +/* The range the helper is faulting in, and the threads that wait for it to + * finish (scr_sa_pf_forget); under the lock. */ +static ScrSaPfRange scr_sa_pf_busy; +static unsigned scr_sa_pf_forgetting; +static pthread_cond_t scr_sa_pf_done = PTHREAD_COND_INITIALIZER; /* 0 = not decided yet, 1 = on, -1 = off (environment, no thread, or a * kernel without MADV_POPULATE_WRITE). */ static _Atomic(int) scr_sa_pf_state; @@ -323,8 +328,9 @@ static void *scr_sa_pf_main(void *arg) { scr_sa_pf_waiting = false; } ScrSaPfRange r = scr_sa_pf_ring[scr_sa_pf_tail++ % SCR_SA_PF_RING]; + scr_sa_pf_busy = r; pthread_mutex_unlock(&scr_sa_pf_lock); - scr_sa_pf_populate(r.lo, r.hi); + if (r.hi > r.lo) scr_sa_pf_populate(r.lo, r.hi); #ifdef SCR_SA_PF_HEAP if (scr_sa_pf_heap_mode) { (void)scr_sa_pf_heap(); @@ -332,10 +338,26 @@ static void *scr_sa_pf_main(void *arg) { } #endif pthread_mutex_lock(&scr_sa_pf_lock); + scr_sa_pf_busy = (ScrSaPfRange){NULL, NULL}; + if (scr_sa_pf_forgetting) pthread_cond_broadcast(&scr_sa_pf_done); } return NULL; } +/* Drops the requests that overlap [lo, hi) and waits for the one the helper + * is faulting in if it overlaps, so pages returned there stay returned. */ +static void scr_sa_pf_forget(char *lo, char *hi) { + pthread_mutex_lock(&scr_sa_pf_lock); + for (unsigned k = scr_sa_pf_tail; k != scr_sa_pf_head; k++) { + ScrSaPfRange *r = &scr_sa_pf_ring[k % SCR_SA_PF_RING]; + if (r->lo < hi && r->hi > lo) r->hi = r->lo; + } + scr_sa_pf_forgetting++; + while (scr_sa_pf_busy.lo < hi && scr_sa_pf_busy.hi > lo) pthread_cond_wait(&scr_sa_pf_done, &scr_sa_pf_lock); + scr_sa_pf_forgetting--; + pthread_mutex_unlock(&scr_sa_pf_lock); +} + static bool scr_sa_pf_on(void) { int s = atomic_load_explicit(&scr_sa_pf_state, memory_order_relaxed); if (s == 0) { @@ -1482,6 +1504,38 @@ void scr_sa_thread_park(void) { #endif } +#if defined(SCR_SA_TRIM) && defined(MADV_DONTNEED) +/* Returns the pages past each slice's frontier that a parking thread leaves + * faulted in but never carved: the rest of the huge page the frontier is in + * (huge pages fault in whole) and the steps faulted ahead of it. The + * thread's objects stay where they are; the next thread to claim the arena + * faults the pages in again as it carves, and they read as zero, as fresh + * pages do. Until then, a program that does its remaining work on threads + * that never claim the arena (parse threads ahead of a check on the main + * thread) does not keep them. */ +static void scr_sa_release_frontiers(void) { + uintptr_t page = (uintptr_t)sysconf(_SC_PAGESIZE); + if (page == 0 || page == (uintptr_t)-1) page = 4096; +#ifdef SCR_SA_THP + const uintptr_t ahead = SCR_SA_HUGE_PAGE; +#else + const uintptr_t ahead = page; +#endif + for (unsigned c = 0; c < SCR_SA_NCLASS; c++) { + if (!scr_sa.lim[c]) continue; + const ScrSaHoles *h = &scr_sa_holes[scr_sa_arena][c]; + char *bump = h->lim ? h->bump : scr_sa.bump[c]; + char *lim = h->lim ? h->lim : scr_sa.lim[c]; + char *end = (char *)scr_sa.base + ((size_t)(c + 1) << scr_sa.shift); + char *from = (char *)(((uintptr_t)bump + page - 1) & ~(page - 1)); + /* At most one step of `ahead` is faulted in past the limit. */ + char *to = (char *)((((uintptr_t)lim + ahead - 1) & ~(ahead - 1)) + ahead); + if (to > end) to = end; + if (to > from) scr_mem_release_ahead(from, to); + } +} +#endif + static void scr_sa_park_arena(void) { if (scr_sa_arena < 0) return; unsigned i = (unsigned)scr_sa_arena; @@ -1503,6 +1557,9 @@ static void scr_sa_park_arena(void) { return; } if (k) scr_sa_keep_merge(k); +#ifdef MADV_DONTNEED + scr_sa_release_frontiers(); +#endif if (scr_sa_trim_worth()) { if (scr_sa_start_helper(i)) { /* Later frees on this thread take the foreign path into the arena's @@ -1748,6 +1805,18 @@ void scr_mem_prefault(void *lo, void *hi) { #endif } +void scr_mem_release_ahead(void *lo, void *hi) { +#if SCR_SMALL_ALLOC && defined(SCR_SA_PREFAULT) + if (atomic_load_explicit(&scr_sa_pf_state, memory_order_relaxed) > 0) scr_sa_pf_forget(lo, hi); +#endif +#if defined(__linux__) && defined(MADV_DONTNEED) + if ((char *)hi > (char *)lo) (void)madvise(lo, (size_t)((char *)hi - (char *)lo), MADV_DONTNEED); +#else + (void)lo; + (void)hi; +#endif +} + /* Free order. Free lists are LIFO, so blocks freed in descending address * order come back in ascending order, the order the bump pointers carve * fresh memory in, and the order the program then touches them in. The diff --git a/packages/runtime/src/scr_alloc.test.c b/packages/runtime/src/scr_alloc.test.c index 042c04c65..2d8024c6d 100644 --- a/packages/runtime/src/scr_alloc.test.c +++ b/packages/runtime/src/scr_alloc.test.c @@ -442,6 +442,68 @@ static void check_trim(void) { } free(trim_blocks); } + +/* A thread that parks returns the pages faulted in past each slice's + * frontier (the rest of the frontier's huge page and the pages faulted in + * ahead), and its blocks stay; the next thread to claim the arena carves on + * from the frontier, zeroed. FRONTIER_SIZE is a class the other checks leave + * unused, and FRONTIER_BYTES carves past SCR_SA_HUGE_AFTER. */ +enum { FRONTIER_SIZE = 120, FRONTIER_BYTES = 12 << 20, FRONTIER_MORE = 4096 }; +static unsigned char **frontier_blocks; +static size_t frontier_n; +static uintptr_t frontier_hi, frontier_end; +static size_t frontier_past; + +static void *frontier_allocate(void *arg) { + (void)arg; + scr_sa_thread_starting(); + for (size_t i = 0; i < frontier_n; i++) { + unsigned char *p = scr_mem_alloc(FRONTIER_SIZE); + assert(p && owned(p)); + fill(p, FRONTIER_SIZE, (unsigned char)i); + frontier_blocks[i] = p; + if ((uintptr_t)p + FRONTIER_SIZE > frontier_hi) frontier_hi = (uintptr_t)p + FRONTIER_SIZE; + } + uintptr_t c = ((uintptr_t)frontier_blocks[0] - scr_sa.base) >> scr_sa.shift; + frontier_end = scr_sa.base + ((c + 1) << scr_sa.shift); + scr_sa_thread_shares(); + scr_sa_thread_park(); + return NULL; +} + +static void *frontier_continue(void *arg) { + (void)arg; + for (size_t i = 0; i < FRONTIER_MORE; i++) { + unsigned char *p = scr_mem_calloc(FRONTIER_SIZE); + assert(p && owned(p)); + for (size_t k = 0; k < FRONTIER_SIZE; k++) assert(p[k] == 0); + if ((uintptr_t)p >= frontier_hi) frontier_past++; + scr_mem_free(p); + } + scr_sa_thread_park(); + return NULL; +} + +static void check_release_frontiers(void) { + frontier_n = FRONTIER_BYTES / FRONTIER_SIZE; + frontier_blocks = malloc(frontier_n * sizeof *frontier_blocks); + assert(frontier_blocks); + pthread_t t; + assert(pthread_create(&t, NULL, frontier_allocate, NULL) == 0); + assert(pthread_join(t, NULL) == 0); + scr_sa_wait_trims(); + size_t page = (size_t)sysconf(_SC_PAGESIZE); + uintptr_t from = (frontier_hi + page - 1) & ~(uintptr_t)(page - 1); + uintptr_t to = from + ((size_t)4 << 20) < frontier_end ? from + ((size_t)4 << 20) : frontier_end; + if (to > from) assert(resident_pages(from, to) == 0); + for (size_t i = 0; i < frontier_n; i++) verify(frontier_blocks[i], FRONTIER_SIZE, (unsigned char)i); + assert(pthread_create(&t, NULL, frontier_continue, NULL) == 0); + assert(pthread_join(t, NULL) == 0); + /* The freed block comes back each time: one block past the frontier. */ + assert(frontier_past >= 1); + for (size_t i = 0; i < frontier_n; i++) scr_mem_free(frontier_blocks[i]); /* remote frees */ + free(frontier_blocks); +} #endif #if defined(SCR_WORKERS) && SCR_SMALL_ALLOC && defined(__linux__) @@ -920,6 +982,7 @@ int main(void) { check_threads(); #ifdef __linux__ check_trim(); + check_release_frontiers(); check_discard(); check_discard_delay_grows(); check_discard_waits_for_loop(); diff --git a/packages/runtime/src/scr_rc.h b/packages/runtime/src/scr_rc.h index 889202975..8fed602c6 100644 --- a/packages/runtime/src/scr_rc.h +++ b/packages/runtime/src/scr_rc.h @@ -273,6 +273,10 @@ void scr_rt_free(void *p); * of [lo, hi), aligned to 256 KiB, in ahead of use; builds without the * helper ignore it. */ void scr_mem_prefault(void *lo, void *hi); +/* Returns the pages of [lo, hi) (page aligned), which hold no data, to the + * system (Linux): pre-fault requests for them are dropped first, so they + * stay returned. Other builds ignore it. */ +void scr_mem_release_ahead(void *lo, void *hi); /* Reorders ptrs, distinct pointers `offset` bytes into blocks that are * about to be freed in array order, so that the free lists hand the blocks * back in ascending address order (scr_alloc.c). Builds without the diff --git a/packages/runtime/src/scr_region.c b/packages/runtime/src/scr_region.c index c9d6c5511..3a7dad541 100644 --- a/packages/runtime/src/scr_region.c +++ b/packages/runtime/src/scr_region.c @@ -1362,6 +1362,18 @@ static void scr_rgn_thread_exit(void) { ScrRegion *r = i < t->nregions ? t->regions[i] : all[0]; if (!r) continue; scr_rgn_spare_drop(r); +#if defined(SCR_WORKERS) && SCR_RGN_MMAP && !defined(SCR_RGN_POISON) + /* Nothing is carved here after a script thread exits: the pages faulted + * in ahead of each chunk's bump go back while the program goes on, and + * its objects stay. */ + if (!scr_context_is_main()) + for (ScrRgnChunk *c = r->chunks; c; c = c->next) { + /* 64 KiB steps cover every page size, as in scr_rgn_chunk_retire. */ + char *from = (char *)(((uintptr_t)c->bump + 0xffff) & ~(uintptr_t)0xffff); + if (c->touched > from) scr_mem_release_ahead(from, c->touched); + c->touched = c->bump; + } +#endif for (ScrRgnSeg *s = r->segs, *next; s; s = next) { next = s->next; free(s); From 237a5111745865115764083495978ae58b50e827 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 09:05:05 -0500 Subject: [PATCH 18/51] Resolve conditional member types over literal class type arguments A generic class instance bound its type parameters only to their mapped IR types, so a literal argument widened (K = "required" became string) and a member typed by a conditional over it, such as `get(): Value` with `type Value = K extends "required" ? T : T | undefined`, refused with SC2001. Class instantiation now also keeps the type arguments as checker types for the members it collects and lowers, as generic functions do. A literal argument for a type parameter that a conditional type in the class depends on is part of the instance key, so Field and Field are separate instances. The conditional evaluation decides `X extends Y` by assignability when both sides are primitive or literal types. Fixes #1240 Co-authored-by: blurname <50226652+blurname@users.noreply.github.com> --- .../src/frontend/lowering/lower-classes.ts | 70 ++++++++++++++++++- .../compiler/src/frontend/lowering/lowerer.ts | 6 +- packages/compiler/src/frontend/type-mapper.ts | 28 +++++++- ...-generic-class-literal-conditional.ts.json | 6 ++ .../6908-generic-class-literal-conditional.ts | 50 +++++++++++++ 5 files changed, 155 insertions(+), 5 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6908-generic-class-literal-conditional.ts.json create mode 100644 tests/corpus/6908-generic-class-literal-conditional.ts diff --git a/packages/compiler/src/frontend/lowering/lower-classes.ts b/packages/compiler/src/frontend/lowering/lower-classes.ts index 371541df8..ae03a748c 100644 --- a/packages/compiler/src/frontend/lowering/lower-classes.ts +++ b/packages/compiler/src/frontend/lowering/lower-classes.ts @@ -688,6 +688,9 @@ export interface ClassInfo { path: InstantiationPath; family: ClassInfo; bindings: Map; + /** The type arguments as checker types, for the resolutions that need + * literal identity (a conditional type over a literal argument). */ + tsBindings: Map; typeArgsText: string; ordinal: number; }; @@ -767,6 +770,9 @@ export interface GenericClassInfo { typeParams: ts.Symbol[]; family: ClassInfo; instances: Map; + /** The type parameters a conditional type in the class depends on + * (literalSensitiveTypeParams), computed on first instantiation. */ + literalSensitive?: Set; } /** The KEY symbol behind a LATE-BOUND (`__@name@id`) property: resolved @@ -1766,6 +1772,7 @@ export function collectClassShapeInner( family: ClassInfo; name: string; bindings: Map; + tsBindings: Map; typeArgsText: string; ordinal: number; }, @@ -4442,6 +4449,7 @@ export function collectClassShapeInner( path: inst.path, family: inst.family, bindings: inst.bindings, + tsBindings: inst.tsBindings, typeArgsText: inst.typeArgsText, ordinal: inst.ordinal, }; @@ -4582,8 +4590,10 @@ export function genericClassInstanceType( .getTypeArguments(ref as ts.TypeReference) .slice(0, gci.typeParams.length); const mapped: IrType[] = []; + const tsArgs: ts.Type[] = []; if (args.length === gci.typeParams.length) { for (const a of args) { + tsArgs.push(lowerer.typeParamTsResolver(a) ?? a); const m = lowerer.mapTypeOf(a, mapArgument); if (!m || m.kind === "void") { // An UNBOUND type parameter argument (`Box` outside any @@ -4602,9 +4612,26 @@ export function genericClassInstanceType( const b = lowerer.typeParamBindings?.get(tp); if (!b) return null; mapped.push(b); + const bound = lowerer.typeParamTsBindings?.get(tp); + if (bound) tsArgs.push(bound); } } - const key = mapped.map(typeKey).join(","); + // A literal argument for a type parameter that a conditional type + // depends on (`K extends "required" ? T : T | undefined`) selects the + // members' types, so it keys its own instance; mapType alone widens it. + gci.literalSensitive ??= literalSensitiveTypeParams(lowerer, gci); + const literalKey = (i: number): string | null => { + if (!gci.literalSensitive!.has(gci.typeParams[i]!)) return null; + const arg = tsArgs[i]; + if (arg?.isStringLiteralType()) return JSON.stringify(arg.value); + if (arg?.isNumberLiteralType()) return String(arg.value); + if (arg && arg.flags & ts.TypeFlags.BooleanLiteral) return lowerer.checker.typeToString(arg); + return null; + }; + const literals = gci.typeParams.map((_, i) => literalKey(i)); + const key = + mapped.map(typeKey).join(",") + + (literals.some((l) => l !== null) ? `@${literals.map((l) => l ?? "*").join(",")}` : ""); const existing = gci.instances.get(key); if (existing) { if (existing.info) lowerer.noteGenericClassInstanceDemand(existing.info); @@ -4626,13 +4653,18 @@ export function genericClassInstanceType( gci.instances.set(key, entry); const bindings = new Map(); gci.typeParams.forEach((tp, i) => bindings.set(tp, mapped[i]!)); - const rendered = mapped.map((m) => lowerer.fmt(m)).join(", "); + const tsBindings = new Map(); + if (tsArgs.length === gci.typeParams.length) + gci.typeParams.forEach((tp, i) => tsBindings.set(tp, tsArgs[i]!)); + const rendered = mapped.map((m, i) => literals[i] ?? lowerer.fmt(m)).join(", "); const typeArgsText = `<${rendered.length > 80 ? rendered.slice(0, 77) + "..." : rendered}>`; const prevBindings = lowerer.typeParamBindings; + const prevTsBindings = lowerer.typeParamTsBindings; const prevContext = lowerer.instantiationContext; const previousPath = lowerer.genericInstantiationPath; lowerer.genericInstantiationPath = path; lowerer.typeParamBindings = bindings; + lowerer.typeParamTsBindings = tsBindings; lowerer.instantiationContext = `instantiating class '${gci.baseName}' with ${typeArgsText}`; try { lowerer.collectClassShapeInner(decl, undefined, { @@ -4640,6 +4672,7 @@ export function genericClassInstanceType( family: gci.family, name, bindings, + tsBindings, typeArgsText, ordinal, }); @@ -4651,6 +4684,7 @@ export function genericClassInstanceType( return null; } finally { lowerer.typeParamBindings = prevBindings; + lowerer.typeParamTsBindings = prevTsBindings; lowerer.instantiationContext = prevContext; lowerer.genericInstantiationPath = previousPath; } @@ -4666,6 +4700,35 @@ export function genericClassInstanceType( return { kind: "object", className: name }; } +/** The class's type parameters that a conditional type inside it depends + * on: the parameters named within a type node that is a conditional type + * or refers to one (`get(): Value` with `type Value = K + * extends "required" ? T : T | undefined`). Their literal arguments pick + * the conditional's branch. */ +function literalSensitiveTypeParams(lowerer: Lowerer, gci: GenericClassInfo): Set { + const params = new Set(gci.typeParams); + const found = new Set(); + const collect = (node: ts.Node): void => { + if (ts.isTypeReferenceNode(node) && ts.isIdentifier(node.typeName)) { + const symbol = lowerer.checker.getSymbolAtLocation(node.typeName); + if (symbol && params.has(symbol)) found.add(symbol); + } + ts.forEachChild(node, collect); + }; + const visit = (node: ts.Node): void => { + if ( + ts.isConditionalTypeNode(node) || + (ts.isTypeReferenceNode(node) && + lowerer.checker.getTypeFromTypeNode(node).isConditionalType()) + ) { + collect(node); + } + ts.forEachChild(node, visit); + }; + for (const member of gci.decl.members.nodes) visit(member); + return found; +} + /** Runs a member-lowering thunk under an INSTANTIATION's type-parameter * bindings (the generic-fn typeParamResolver mechanism) — the checker * keeps reporting the unsubstituted `T`s inside the shared body AST. @@ -4699,17 +4762,20 @@ function withInstanceBindings(lowerer: Lowerer, info: ClassInfo, fn: () => T) } } const prevBindings = lowerer.typeParamBindings; + const prevTsBindings = lowerer.typeParamTsBindings; const prevContext = lowerer.instantiationContext; const prevSuppress = lowerer.suppressStats; const previousPath = lowerer.genericInstantiationPath; lowerer.genericInstantiationPath = gi.path; lowerer.typeParamBindings = gi.bindings; + lowerer.typeParamTsBindings = gi.tsBindings; lowerer.instantiationContext = `instantiating class '${gi.family.generic?.baseName ?? info.def.jsName ?? ""}' with ${gi.typeArgsText}`; lowerer.suppressStats = prevSuppress || gi.ordinal > 0; try { return fn(); } finally { lowerer.typeParamBindings = prevBindings; + lowerer.typeParamTsBindings = prevTsBindings; lowerer.instantiationContext = prevContext; lowerer.suppressStats = prevSuppress; lowerer.genericInstantiationPath = previousPath; diff --git a/packages/compiler/src/frontend/lowering/lowerer.ts b/packages/compiler/src/frontend/lowering/lowerer.ts index 543149f83..d3ea6aa2d 100644 --- a/packages/compiler/src/frontend/lowering/lowerer.ts +++ b/packages/compiler/src/frontend/lowering/lowerer.ts @@ -8644,8 +8644,9 @@ export class Lowerer { /** The ts-level twin of typeParamResolver: the bound CHECKER type of a * type parameter in the current instantiation (typeParamTsBindings), for * the resolutions where mapType's widening already dropped what the body - * needs (indexed accesses over literal-bound keys). Inert outside - * call-keyed generic instantiation. */ + * needs (indexed accesses over literal-bound keys, conditional types + * over literal arguments). Inert outside generic function and class + * instantiation. */ readonly typeParamTsResolver = (t: ts.Type): ts.Type | null => { if (!this.typeParamTsBindings || !(t.flags & ts.TypeFlags.TypeParameter)) return null; const sym: ts.Symbol | undefined = t.getSymbol(); @@ -10838,6 +10839,7 @@ export class Lowerer { family: ClassInfo; name: string; bindings: Map; + tsBindings: Map; typeArgsText: string; ordinal: number; }, diff --git a/packages/compiler/src/frontend/type-mapper.ts b/packages/compiler/src/frontend/type-mapper.ts index ca838cea2..3448545f8 100644 --- a/packages/compiler/src/frontend/type-mapper.ts +++ b/packages/compiler/src/frontend/type-mapper.ts @@ -5836,7 +5836,9 @@ function evalMappedTemplate( if (!checkType || !extendsType) return null; const checked = evalMappedTemplate(checkType, keys, scope, ctx); if (!checked) return null; - const inferred = matchInferReference(checked, extendsType, ctx); + const inferred = + primitiveExtendsCheck(checked, extendsType, ctx) ?? + matchInferReference(checked, extendsType, ctx); if (inferred === undefined) return null; const branch = inferred ? type.getTrueType() : type.getFalseType(); const branchScope = inferred ? new Map([...scope, ...inferred]) : scope; @@ -5845,6 +5847,30 @@ function evalMappedTemplate( return mentionsMappedParams(type, keys, aliases) ? null : type; } +/** `checked extends X` where X is a primitive or literal type, or a union of + * them (`K extends "required" ? ...`), and `checked` is one such type: + * X has no infer positions, so assignability decides the branch (an empty + * binding map for the true branch). A union `checked` distributes and stays + * undecided here (undefined), as do other shapes. */ +function primitiveExtendsCheck( + checked: ts.Type, + extendsType: ts.Type, + ctx: TypeMapperCtx, +): Map | false | undefined { + const primitive = + ts.TypeFlags.StringLike | + ts.TypeFlags.NumberLike | + ts.TypeFlags.BooleanLike | + ts.TypeFlags.BigIntLike | + ts.TypeFlags.Null | + ts.TypeFlags.Undefined; + const isPrimitive = (t: ts.Type): boolean => + !t.isUnionType() && (t.flags & primitive) !== 0 && (t.flags & ts.TypeFlags.TypeParameter) === 0; + const extendsParts = extendsType.isUnionType() ? ts.constituentTypes(extendsType) : [extendsType]; + if (!isPrimitive(checked) || !extendsParts.every(isPrimitive)) return undefined; + return ctx.checker.isTypeAssignableTo(checked, extendsType) ? new Map() : false; +} + /** `checked extends G<..., infer T, ...>`: the infer bindings when * `checked` is a reference to the same generic `G` (its arguments bind the * infer positions; other positions must be the same types), false when it diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6908-generic-class-literal-conditional.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6908-generic-class-literal-conditional.ts.json new file mode 100644 index 000000000..d0403f33f --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6908-generic-class-literal-conditional.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6908-generic-class-literal-conditional.ts" + ], + "diags": [] +} diff --git a/tests/corpus/6908-generic-class-literal-conditional.ts b/tests/corpus/6908-generic-class-literal-conditional.ts new file mode 100644 index 000000000..d06669be3 --- /dev/null +++ b/tests/corpus/6908-generic-class-literal-conditional.ts @@ -0,0 +1,50 @@ +// A generic class whose member types are conditional types over a literal +// type parameter: each literal argument selects the conditional's branch, +// so Field.get() returns T and Field.get() +// returns T | undefined. +type Kind = "required" | "optional"; +type Value = K extends "required" ? T : T | undefined; + +class Field { + value: T; + kind: K; + present: boolean; + constructor(value: T, kind: K, present = true) { + this.value = value; + this.kind = kind; + this.present = present; + } + get(): Value { + return (this.present ? this.value : undefined) as Value; + } + describe(): string { + return `${this.kind}:${String(this.get())}`; + } +} + +console.log(new Field("hello", "required").get()); +const name = new Field("ada", "required"); +const nick = new Field("ad", "optional", false); +const age = new Field(36, "optional"); +console.log(name.get().toUpperCase(), nick.get() ?? "(none)", (age.get() ?? 0) + 1); +console.log(name.describe(), nick.describe(), age.describe()); + +function read(field: Field): string { + return field.get(); +} +function readOptional(field: Field): string { + return field.get() ?? "missing"; +} +console.log(read(name), readOptional(nick), readOptional(new Field("x", "optional"))); + +type Flag = B extends true ? "on" : "off"; +class Switch { + readonly state: B; + constructor(state: B) { + this.state = state; + } + label(): Flag { + return (this.state ? "on" : "off") as Flag; + } +} +console.log(new Switch(true).label(), new Switch(false).label()); From 050bbd852c15a2cf720b987bbb58c093b0b3836c Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 09:09:19 -0500 Subject: [PATCH 19/51] Collect the frontend's cycles when the native compiler disposes it The frontend's syntax trees and program state refer to each other, so reference counting frees almost none of them when a build disposes the frontend. Whether a cycle pass reclaimed them before the build ended depended on where the collector's schedule happened to be. Passes during the frontend found its data live and widened the growth fractions, so the next pass waited until the heap had grown well past its size at disposal, and the dead frontend and the emitter's garbage stayed until the build ended. The native compiler now runs scr_collect_cycles_phase right after it disposes the frontend, through a host function. The runtime entry point sweeps, then restarts the growth fractions and the death rule's backoff from the heap the sweep leaves, as after a productive pass. Under Node the hook is unset. --- packages/compiler/native/host.c | 12 ++++++++++ packages/compiler/native/host.ffi.json | 3 ++- packages/compiler/src/executable/prepare.ts | 2 ++ .../compiler/src/frontend/dispose-hook.ts | 14 +++++++++++ packages/compiler/src/library/prepare.ts | 2 ++ packages/compiler/src/native/cli.ts | 3 +++ packages/runtime/src/scr_cycle.c | 17 +++++++++++++ packages/runtime/src/scr_rc.h | 1 + packages/runtime/test/test_cycle.c | 24 +++++++++++++++++++ 9 files changed, 77 insertions(+), 1 deletion(-) create mode 100644 packages/compiler/src/frontend/dispose-hook.ts diff --git a/packages/compiler/native/host.c b/packages/compiler/native/host.c index 75c2dcbaf..6aa1b2a28 100644 --- a/packages/compiler/native/host.c +++ b/packages/compiler/native/host.c @@ -44,3 +44,15 @@ uint8_t scriptc_native_private_directory(const uint8_t *bytes, size_t size, uint return ok; #endif } + +/* The runtime's full sweep after a phase's data died (scr_rc.h). */ +void scr_collect_cycles_phase(void); + +/* Reclaims the cycles a finished compiler phase leaves behind. The + * frontend's syntax trees and program state refer to each other, so + * reference counting frees almost none of them when the frontend is + * disposed, and the collector's schedule alone may not run a full pass + * before the build ends. */ +void scriptc_native_collect_cycles(void) { + scr_collect_cycles_phase(); +} diff --git a/packages/compiler/native/host.ffi.json b/packages/compiler/native/host.ffi.json index f2a738054..d7e0e92aa 100644 --- a/packages/compiler/native/host.ffi.json +++ b/packages/compiler/native/host.ffi.json @@ -1,6 +1,7 @@ { "ffi_format": 6, "functions": [ - { "name": "compilerNativePrivateDirectory", "symbol": "scriptc_native_private_directory", "params": ["string", "bool"], "returns": "bool" } + { "name": "compilerNativePrivateDirectory", "symbol": "scriptc_native_private_directory", "params": ["string", "bool"], "returns": "bool" }, + { "name": "compilerNativeCollectCycles", "symbol": "scriptc_native_collect_cycles", "params": [], "returns": "void" } ] } diff --git a/packages/compiler/src/executable/prepare.ts b/packages/compiler/src/executable/prepare.ts index d97497c70..4afae7c7d 100644 --- a/packages/compiler/src/executable/prepare.ts +++ b/packages/compiler/src/executable/prepare.ts @@ -1,5 +1,6 @@ import type { CompileFailure, CompileRequestOptions } from "../compile-types.js"; import type { FfiProfile } from "../ffi/ffi-manifest.js"; +import { frontendDisposed } from "../frontend/dispose-hook.js"; import { checkerPanicDiag, npmStaticRefusalDiag, @@ -112,6 +113,7 @@ export function prepareExecutableModule( } } finally { fe.dispose(); + frontendDisposed(); timing("frontend-dispose"); } diff --git a/packages/compiler/src/frontend/dispose-hook.ts b/packages/compiler/src/frontend/dispose-hook.ts new file mode 100644 index 000000000..ccc9b19aa --- /dev/null +++ b/packages/compiler/src/frontend/dispose-hook.ts @@ -0,0 +1,14 @@ +/* What runs right after a build disposes its frontend. The native compiler + * sets it to a full cycle collection (native/cli.ts): the frontend's syntax + * trees and program state refer to each other, so disposing them leaves + * cyclic garbage that reference counting cannot free. Under Node it is + * unset. */ +let afterDispose: (() => void) | null = null; + +export function setFrontendDisposedHook(hook: () => void): void { + afterDispose = hook; +} + +export function frontendDisposed(): void { + if (afterDispose !== null) afterDispose(); +} diff --git a/packages/compiler/src/library/prepare.ts b/packages/compiler/src/library/prepare.ts index 298bb43bb..fdaf8f506 100644 --- a/packages/compiler/src/library/prepare.ts +++ b/packages/compiler/src/library/prepare.ts @@ -1,5 +1,6 @@ import { dirname, resolve } from "node:path"; import type { CompileFailure } from "../compile-types.js"; +import { frontendDisposed } from "../frontend/dispose-hook.js"; import { fenceSpeculativeWasiFunctions, moduleWasiUnavailableSurface, @@ -585,6 +586,7 @@ export function prepareLibrary( sourceTexts = fe.sourceTexts(); } finally { fe.dispose(); + frontendDisposed(); } const mod = lowered.module!; timing("frontend-dispose"); diff --git a/packages/compiler/src/native/cli.ts b/packages/compiler/src/native/cli.ts index d7520ae0d..ed8d52293 100644 --- a/packages/compiler/src/native/cli.ts +++ b/packages/compiler/src/native/cli.ts @@ -10,6 +10,7 @@ import { } from "../backend/runtime-pack-native.js"; import { mergeNativeProfiles } from "../backend/native-tools.js"; import { setDeclarationRoot } from "../frontend/dts-paths.js"; +import { setFrontendDisposedHook } from "../frontend/dispose-hook.js"; import { FrontendServices } from "../frontend/services.js"; import { createNativeTs7Api } from "../frontend/ts7/native-api.js"; import { resolveProvenanceSourcesWithParser } from "../frontend/provenance-core.js"; @@ -21,8 +22,10 @@ import { buildSanitizedRuntime } from "./sanitizer.js"; import { runCompilerTask } from "./task.js"; declare function compilerNativePrivateDirectory(path: string, harden: boolean): boolean; +declare function compilerNativeCollectCycles(): void; async function main(): Promise { + setFrontendDisposedHook(() => compilerNativeCollectCycles()); const manifestPath = process.env["SCRIPTC_TOOLCHAIN"] ?? process.execPath + ".json"; let compiler: NativeCompiler | null = null; let cache: ReturnType = null; diff --git a/packages/runtime/src/scr_cycle.c b/packages/runtime/src/scr_cycle.c index 93180deca..ce4dfa5fa 100644 --- a/packages/runtime/src/scr_cycle.c +++ b/packages/runtime/src/scr_cycle.c @@ -2195,6 +2195,23 @@ void scr_collect_cycles(void) { } } +/* The full sweep, for a caller that knows a phase's data has just died (a + * compiler dropping its frontend). Such data is mostly cyclic, so reference + * counting frees little of it and the schedule cannot see it die. The + * growth fractions the phase widened, while its live data kept passes from + * finding anything, and the death rule's backoff then start over from the + * heap the sweep leaves, as after a productive pass. Otherwise the next + * phase's garbage would accumulate until the heap grew to that widened + * fraction. */ +void scr_collect_cycles_phase(void) { + scr_collect_cycles(); + for (unsigned gen = SCR_CYC_MATURE; gen < SCR_CYC_NGENS; gen++) { + scr_cyc_growth_shift[gen] = 0; + scr_cyc_growth_streak[gen] = 0; + } + scr_cyc_death_shift = 0; +} + /* Worker TLS becomes unreachable when its OS thread returns. Release the * reusable traversal buffers after all context-owned roots are retired. */ void scr_cyc_context_cleanup(void) { diff --git a/packages/runtime/src/scr_rc.h b/packages/runtime/src/scr_rc.h index 889202975..39e962512 100644 --- a/packages/runtime/src/scr_rc.h +++ b/packages/runtime/src/scr_rc.h @@ -601,6 +601,7 @@ static inline void scr_cyc_dead(void *obj) { * after and wants nothing reclaimable left), and it costs a walk of the * live heap — do not put it on a per-turn path. */ void scr_collect_cycles(void); +void scr_collect_cycles_phase(void); void scr_cyc_context_cleanup(void); /* One pass on the normal generational schedule, for callers that reach a diff --git a/packages/runtime/test/test_cycle.c b/packages/runtime/test/test_cycle.c index d4deec3c3..723e9519f 100644 --- a/packages/runtime/test/test_cycle.c +++ b/packages/runtime/test/test_cycle.c @@ -536,6 +536,29 @@ static void check_mixed_dead_structure_narrows(void) { check(live <= narrowed, "a pass that freed a dead mature and old structure did not narrow the growth fraction"); } +/* The phase sweep (scr_collect_cycles_phase) is for a caller that knows a + * phase's data has just died: it sweeps, and the growth schedule starts + * over. After passes that find nothing have widened the old level's + * fraction to the cap, the plain sweep leaves the next dead cycle to the + * cap, while after the phase sweep it goes once the heap has grown the base + * quarter past the floor. */ +static void check_phase_sweep_restarts_schedule(void) { + enum { FLOOR = 4096 }; + size_t cap = configured_growth_cap(); + if (cap < 2) return; /* the base fraction is already the cap */ + size_t slack = 2 * configured_nursery_threshold() + 2; + size_t due = FLOOR + FLOOR * cap; + size_t live = 0; + for (int i = 0; i < 4; i++) live = grow_dead_cycle_until_full_pass(0, false); + check(live > due, "passes that found nothing did not widen the growth fraction to the cap"); + scr_collect_cycles(); + live = grow_dead_cycle_until_full_pass(0, false); + check(live > due, "the plain sweep changed the growth schedule"); + scr_collect_cycles_phase(); + live = grow_dead_cycle_until_full_pass(0, false); + check(live <= FLOOR + FLOOR / 4 + slack, "the phase sweep did not restart the growth schedule"); +} + static void check_deep_ring(void) { enum { DEPTH = 100000 }; size_t before = freed; @@ -1433,6 +1456,7 @@ int main(void) { check_growth_cap_bounds_floating_garbage(); check_mixed_dead_structure_narrows(); check_death_waits_for_idle(); + check_phase_sweep_restarts_schedule(); check_death_brings_full_pass(); check_nursery_bypass(); check_mature_bypass(); From fdc6c4ec5d6d0b2c94390de96f12a81e4ef46182 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 09:11:25 -0500 Subject: [PATCH 20/51] Convert string index arguments without libm calls The runtime's ToIntegerOrInfinity for string index arguments (slice, substring, charAt, indexOf, startsWith, padStart, repeat and the others in scr_string.c) converts through int64 instead of calling trunc, which is a libm call on baseline x86-64. Each slice made two such calls. --- packages/runtime/src/scr_string.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/packages/runtime/src/scr_string.c b/packages/runtime/src/scr_string.c index ddd984e72..41588f25a 100644 --- a/packages/runtime/src/scr_string.c +++ b/packages/runtime/src/scr_string.c @@ -1759,10 +1759,14 @@ static __attribute__((noinline)) size_t scr_byte_to_u16_walk(const ScrStr *s, Sc } /* ECMA-262 ToIntegerOrInfinity for a double already known to be a Number: - * NaN → +0, otherwise truncate toward zero, ±Infinity preserved. */ -static double scr_to_integer_or_infinity(double x) { - if (isnan(x)) return 0.0; - return trunc(x); + * NaN → +0, otherwise truncate toward zero, ±Infinity preserved. Inside + * (-2^63, 2^63) the integer conversion truncates, without the libm trunc + * call baseline x86-64 needs; a fraction above -1 gives +0, as the spec's + * mathematical result does. Every double outside that range is already an + * integer or infinite, and NaN fails both comparisons. */ +static inline double scr_to_integer_or_infinity(double x) { + if (x > -9223372036854775808.0 && x < 9223372036854775808.0) return (double)(int64_t)x; + return isnan(x) ? 0.0 : x; } /* Byte substring search (needle and hay are both canonical WTF-8, From 973884eabe00ddc4e0e18cb0cc41e8d88bbf35b8 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 09:13:12 -0500 Subject: [PATCH 21/51] Support fs.promises.rm options fs.promises.rm(path, options) accepts the { recursive, force, maxRetries, retryDelay } literal options that rmSync accepts. It runs the same removal and rejects with its errors. Removing a directory without recursive now throws Node's ERR_FS_EISDIR SystemError from rmSync and fs.promises.rm. A retryDelay outside int32 >= 0 or a maxRetries outside uint32 throws ERR_OUT_OF_RANGE. rmSync checks these options before the path, and fs.promises.rm checks the path first, as Node does. Fixes #1615 Co-authored-by: jlucaso1 <55464917+jlucaso1@users.noreply.github.com> --- .../ambient/scriptc-node-fallback.d.ts | 4 + .../compiler/src/backend/llvm/lib-shared.ts | 2 + .../lowering/builtins/filesystem-calls.ts | 28 ++++-- .../lowering/builtins/module-calls.ts | 5 +- .../src/frontend/lowering/surfaces.ts | 2 + .../compiler/src/ir/builtin-signatures.ts | 14 ++- .../6930-fs-promises-rm-options.ts.json | 6 ++ packages/runtime/src/scr_async.c | 13 +++ packages/runtime/src/scr_lib.c | 42 +++++--- packages/runtime/src/scr_lib.h | 4 + packages/runtime/test/lib.test.ts | 5 +- tests/corpus/6930-fs-promises-rm-options.ts | 99 +++++++++++++++++++ .../6930-fs-promises-rm-options.ts.json | 1 + 13 files changed, 196 insertions(+), 29 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6930-fs-promises-rm-options.ts.json create mode 100644 tests/corpus/6930-fs-promises-rm-options.ts create mode 100644 tests/harness/corpus-costs/6930-fs-promises-rm-options.ts.json diff --git a/packages/compiler/ambient/scriptc-node-fallback.d.ts b/packages/compiler/ambient/scriptc-node-fallback.d.ts index fb107ef99..329f066fc 100644 --- a/packages/compiler/ambient/scriptc-node-fallback.d.ts +++ b/packages/compiler/ambient/scriptc-node-fallback.d.ts @@ -2128,6 +2128,10 @@ declare module "fs/promises" { options: import("node:fs").GlobOptionsWithFileTypes, ): AsyncIterableIterator; export function rm(path: string): Promise; + export function rm( + path: string, + options: { recursive?: boolean; force?: boolean; maxRetries?: number; retryDelay?: number }, + ): Promise; export function stat(path: string): Promise; export function statfs( path: string | Uint8Array | URL, diff --git a/packages/compiler/src/backend/llvm/lib-shared.ts b/packages/compiler/src/backend/llvm/lib-shared.ts index 76b6e88f5..ffce524d8 100644 --- a/packages/compiler/src/backend/llvm/lib-shared.ts +++ b/packages/compiler/src/backend/llvm/lib-shared.ts @@ -793,6 +793,8 @@ export const LIB_FN_SYMS: Record = { "fsp.rename": "scr_fsp_rename", "fsp.readdir": "scr_fsp_readdir", "fsp.rm": "scr_fsp_rm", + "fsp.rmOpts": "scr_fsp_rm_opts", + "fsp.rmRetry": "scr_fsp_rm_opts_retry", "fsp.stat": "scr_fsp_stat", "fsp.realpath": "scr_fsp_realpath", "fsp.lstat": "scr_fsp_lstat", diff --git a/packages/compiler/src/frontend/lowering/builtins/filesystem-calls.ts b/packages/compiler/src/frontend/lowering/builtins/filesystem-calls.ts index 05028e1df..4de69a3f6 100644 --- a/packages/compiler/src/frontend/lowering/builtins/filesystem-calls.ts +++ b/packages/compiler/src/frontend/lowering/builtins/filesystem-calls.ts @@ -863,15 +863,22 @@ export function lowerFsPromisesMkdirCall( : { kind: "libCall", fn: "fsp.mkdirRecursive", args: [path], type, loc }; } -// rmSync(p, options): literal { recursive?, force? } booleans — the -// cleanup shape rmSync(dir, { recursive: true, force: true }) — -// plus the maxRetries/retryDelay numbers (the tmpdir-harness shape +// rmSync(p, options) and fs.promises.rm(p, options): literal +// { recursive?, force? } booleans — the cleanup shape +// rmSync(dir, { recursive: true, force: true }) — plus the +// maxRetries/retryDelay numbers (the tmpdir-harness shape // rmSync(p, { maxRetries: 3, recursive: true, force: true })): those // lower as ordinary number expressions into the retry libCall, whose // runtime implements Node's linear-backoff retry on // EBUSY/EMFILE/ENFILE/ENOTEMPTY/EPERM. The retry-free shape keeps the -// historical rmOptsSync lowering unchanged. -export function lowerFsRemoveCall(lowerer: Lowerer, expr: ts.CallExpression, loc: SrcLoc): IrExpr { +// historical rmOptsSync lowering unchanged. The promise form runs the same +// removal and rejects with its error. +export function lowerFsRemoveCall( + lowerer: Lowerer, + expr: ts.CallExpression, + loc: SrcLoc, + promise: boolean, +): IrExpr { const opts = literalBoolOptions( expr.arguments.nodes[1]!, ["recursive", "force"], @@ -879,7 +886,7 @@ export function lowerFsRemoveCall(lowerer: Lowerer, expr: ts.CallExpression, loc ); if (opts === null) { lowerer.noLowering( - "rmSync with an options argument beyond { recursive, force, maxRetries, retryDelay }", + `${promise ? "fs.promises.rm" : "rmSync"} with an options argument beyond { recursive, force, maxRetries, retryDelay }`, expr.arguments.nodes[1]!, "the recursive/force flags must be boolean literals; maxRetries/retryDelay are numbers; other options have no lowering", ); @@ -888,12 +895,13 @@ export function lowerFsRemoveCall(lowerer: Lowerer, expr: ts.CallExpression, loc const recursive = boolLit(opts.bools["recursive"] === true, loc); const force = boolLit(opts.bools["force"] === true, loc); + const type: IrType = promise ? { kind: "promise", inner: VOID } : VOID; if (opts.exprs["maxRetries"] === undefined && opts.exprs["retryDelay"] === undefined) { return { kind: "libCall", - fn: "fs.rmOptsSync", + fn: promise ? "fsp.rmOpts" : "fs.rmOptsSync", args: [path, recursive, force], - type: VOID, + type, loc, }; } @@ -904,7 +912,7 @@ export function lowerFsRemoveCall(lowerer: Lowerer, expr: ts.CallExpression, loc node ? lowerer.lowerExprExpecting(node, F64) : { kind: "numLit", value: dflt, type: F64, loc }; return { kind: "libCall", - fn: "fs.rmRetrySync", + fn: promise ? "fsp.rmRetry" : "fs.rmRetrySync", args: [ path, recursive, @@ -912,7 +920,7 @@ export function lowerFsRemoveCall(lowerer: Lowerer, expr: ts.CallExpression, loc num(opts.exprs["maxRetries"], 0), num(opts.exprs["retryDelay"], 100), ], - type: VOID, + type, loc, }; } diff --git a/packages/compiler/src/frontend/lowering/builtins/module-calls.ts b/packages/compiler/src/frontend/lowering/builtins/module-calls.ts index c85f2c777..970a474c7 100644 --- a/packages/compiler/src/frontend/lowering/builtins/module-calls.ts +++ b/packages/compiler/src/frontend/lowering/builtins/module-calls.ts @@ -292,7 +292,10 @@ export function lowerBuiltinModuleCall( return lowerFsPromisesMkdirCall(lowerer, expr, loc); } if (bi.module === "fs" && bi.member === "rmSync" && expr.arguments.nodes.length === 2) { - return lowerFsRemoveCall(lowerer, expr, loc); + return lowerFsRemoveCall(lowerer, expr, loc, false); + } + if (bi.module === "fs/promises" && bi.member === "rm" && expr.arguments.nodes.length === 2) { + return lowerFsRemoveCall(lowerer, expr, loc, true); } // accessSync(p, mode?) and truncateSync(p, len?), with their promise // spellings: an omitted mode is Node's F_OK (0) and an omitted len is 0. diff --git a/packages/compiler/src/frontend/lowering/surfaces.ts b/packages/compiler/src/frontend/lowering/surfaces.ts index 4e08ba401..c77938337 100644 --- a/packages/compiler/src/frontend/lowering/surfaces.ts +++ b/packages/compiler/src/frontend/lowering/surfaces.ts @@ -2104,6 +2104,8 @@ export const BUILTIN_MODULE_FN_ALIASES: Record< glob: ["fsp.globTypes"], // The Uint8Array/Buffer-data form. appendFile: ["fsp.appendFileBytes"], + // The options forms. + rm: ["fsp.rmOpts", "fsp.rmRetry"], }, crypto: { // The composed randomBytes(n).toString(enc) chain keeps its one-libCall diff --git a/packages/compiler/src/ir/builtin-signatures.ts b/packages/compiler/src/ir/builtin-signatures.ts index eb2718616..ffe91a6b7 100644 --- a/packages/compiler/src/ir/builtin-signatures.ts +++ b/packages/compiler/src/ir/builtin-signatures.ts @@ -3139,6 +3139,13 @@ export const LIB_FN_SIGS = defineLibFnSignatures({ * scandir failure becomes its rejection. */ "fsp.readdirTypes": { argTypes: [STRING], result: { kind: "promise", inner: VOID } }, "fsp.rm": { argTypes: [STRING], result: { kind: "promise", inner: VOID } }, + /** `fs.promises.rm(path, options)`: fs.rmOptsSync and fs.rmRetrySync + * behind a settled promise; their errors become its rejection. */ + "fsp.rmOpts": { argTypes: [STRING, BOOL, BOOL], result: { kind: "promise", inner: VOID } }, + "fsp.rmRetry": { + argTypes: [STRING, BOOL, BOOL, F64, F64], + result: { kind: "promise", inner: VOID }, + }, "fsp.stat": { argTypes: [STRING], result: { kind: "promise", inner: STATS_T } }, "fsp.realpath": { argTypes: [STRING], result: { kind: "promise", inner: STRING } }, "fsp.lstat": { argTypes: [STRING], result: { kind: "promise", inner: STATS_T } }, @@ -4622,8 +4629,7 @@ export const LIB_FN_SIGS = defineLibFnSignatures({ * return value has no lowering (statement position only, frontend- * fenced). rmOptsSync is rmSync with (recursive, force) bools: force * swallows ENOENT, recursive removes trees post-order, a directory - * without recursive throws the EISDIR-worded error (divergence 13's - * wording note). mkdtempSync appends the six X's and returns the + * without recursive throws Node's ERR_FS_EISDIR SystemError. mkdtempSync appends the six X's and returns the * created path (+1). accessSync takes the F_OK/R_OK/W_OK/X_OK bits as * one f64 (the frontend bakes fs.constants.* as literals and completes * an omitted mode to 0). readFdSync/readFdSyncBytes are the @@ -4634,7 +4640,9 @@ export const LIB_FN_SIGS = defineLibFnSignatures({ /** rmOptsSync's maxRetries/retryDelay form (path, recursive, force, * maxRetries, retryDelay): Node's linear-backoff retry on * EBUSY/EMFILE/ENFILE/ENOTEMPTY/EPERM — the tmpdir-harness shape - * rmSync(p, { maxRetries: 3, recursive: true, force: true }). */ + * rmSync(p, { maxRetries: 3, recursive: true, force: true }). Throws + * ERR_OUT_OF_RANGE for a retryDelay outside int32 >= 0 or a maxRetries + * outside uint32, before checking the path. */ "fs.rmRetrySync": { argTypes: [STRING, BOOL, BOOL, F64, F64], result: VOID }, "fs.mkdtempSync": { argTypes: [STRING], result: STRING }, "fs.accessSync": { argTypes: [STRING, F64], result: VOID }, diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6930-fs-promises-rm-options.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6930-fs-promises-rm-options.ts.json new file mode 100644 index 000000000..b4ade9601 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6930-fs-promises-rm-options.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6930-fs-promises-rm-options.ts" + ], + "diags": [] +} diff --git a/packages/runtime/src/scr_async.c b/packages/runtime/src/scr_async.c index 6055734b3..12cf81f77 100644 --- a/packages/runtime/src/scr_async.c +++ b/packages/runtime/src/scr_async.c @@ -2900,6 +2900,19 @@ ScrPromise *scr_fsp_rm(ScrStr *path) { return scr_promise_settled_void(); } +ScrPromise *scr_fsp_rm_opts(ScrStr *path, bool recursive, bool force) { + scr_fs_rm_opts(path, recursive, force); + return scr_promise_settled_void(); +} + +/* fs.promises.rm validates the path before its options, unlike rmSync. */ +ScrPromise *scr_fsp_rm_opts_retry(ScrStr *path, bool recursive, bool force, double max_retries, + double retry_delay) { + if (scr_fs_path_ok(path, "path") && scr_fs_rm_retry_options_ok(max_retries, retry_delay)) + scr_fs_rm_opts_retry(path, recursive, force, max_retries, retry_delay); + return scr_promise_settled_void(); +} + ScrPromise *scr_fsp_stat(ScrStr *path) { ScrStats *st = scr_fs_stat(path); return scr_promise_settled_ref(st, &scr_stats_retain_v, &scr_stats_release_v, NULL); diff --git a/packages/runtime/src/scr_lib.c b/packages/runtime/src/scr_lib.c index 976847049..db0926690 100644 --- a/packages/runtime/src/scr_lib.c +++ b/packages/runtime/src/scr_lib.c @@ -4596,6 +4596,13 @@ void scr_fs_rename(ScrStr *oldpath, ScrStr *newpath) { static int scr_rm_unlink(const char *path, size_t len); +/* Node's ERR_FS_EISDIR for rm on a directory without `recursive`. */ +static void scr_fs_rm_throw_eisdir(const ScrStr *path) { + ScrStr *message = scr_str_new("is a directory", 14); + scr_fs_throw_system_error("ERR_FS_EISDIR", "Path is a directory", EISDIR, "rm", message, path); + scr_str_release(message); +} + void scr_fs_rm(ScrStr *path) { if (!scr_fs_path_ok(path, "path")) return; #ifdef _WIN32 @@ -4603,16 +4610,15 @@ void scr_fs_rm(ScrStr *path) { if (link != 0) { if (link < 0) scr_fs_throw(errno, "unlink", path); return; } #endif /* Node's rmSync: lstat first (a missing path reports the lstat syscall), - * refuse directories (Node requires `recursive`, which the scriptc - * surface doesn't declare — the message wording diverges from Node's - * ERR_FS_EISDIR, see SEMANTICS.md), then unlink. */ + * refuse directories (ERR_FS_EISDIR: removing one requires `recursive`), + * then unlink. */ struct stat st; if (lstat(path->data, &st) != 0) { scr_fs_throw(errno, "lstat", path); return; } if (S_ISDIR(st.st_mode)) { - scr_fs_throw(EISDIR, "rm", path); + scr_fs_rm_throw_eisdir(path); return; } if (scr_rm_unlink(path->data, path->len) != 0) scr_fs_throw(errno, "unlink", path); @@ -4766,6 +4772,13 @@ typedef struct { ScrStr *path; } ScrRmFail; +/* Throws the recorded failure and releases its path. */ +static void scr_rm_fail_throw(ScrRmFail *f) { + if (f->err == EISDIR && strcmp(f->op, "rm") == 0) scr_fs_rm_throw_eisdir(f->path); + else scr_fs_throw(f->err, f->op, f->path); + scr_str_release(f->path); +} + static void scr_rm_fail_set(ScrRmFail *f, int err, const char *op, const char *path, size_t len) { if (f->err != 0) return; /* first failure wins, exactly like the old unwind */ f->err = err; @@ -4861,8 +4874,6 @@ static void scr_fs_rm_attempt(ScrStr *path, bool recursive, bool force, ScrRmFai } if (S_ISDIR(st.st_mode)) { if (!recursive) { - /* Node throws ERR_FS_EISDIR here; the EISDIR-prefixed wording is - * divergence 13's documented difference. */ scr_rm_fail_set(f, EISDIR, "rm", path->data, path->len); return; } @@ -4876,18 +4887,24 @@ void scr_fs_rm_opts(ScrStr *path, bool recursive, bool force) { if (!scr_fs_path_ok(path, "path")) return; ScrRmFail f = {0, NULL, NULL}; scr_fs_rm_attempt(path, recursive, force, &f); - if (f.err != 0) { - scr_fs_throw(f.err, f.op, f.path); - scr_str_release(f.path); - } + if (f.err != 0) scr_rm_fail_throw(&f); +} + +/* Node's validateRmdirOptions range checks: retryDelay is an int32 >= 0, + * then maxRetries is a uint32 (ERR_OUT_OF_RANGE). */ +bool scr_fs_rm_retry_options_ok(double max_retries, double retry_delay) { + return scr_bytes_validate_off("options.retryDelay", retry_delay, 2147483647.0) && + scr_bytes_validate_off("options.maxRetries", max_retries, 4294967295.0); } /* rmSync(p, { recursive, force, maxRetries, retryDelay }): Node retries * the operation on EBUSY/EMFILE/ENFILE/ENOTEMPTY/EPERM up to maxRetries * times, waiting retryDelay ms LONGER on each try (linear backoff — the * fs.rmSync documented semantics). Everything else throws immediately - * with the failing path, exactly like the plain options form. */ + * with the failing path, exactly like the plain options form. rmSync + * validates the options before the path. */ void scr_fs_rm_opts_retry(ScrStr *path, bool recursive, bool force, double max_retries, double retry_delay) { + if (!scr_fs_rm_retry_options_ok(max_retries, retry_delay)) return; if (!scr_fs_path_ok(path, "path")) return; long tries = max_retries > 0 ? (long)max_retries : 0; double delay_ms = retry_delay > 0 ? retry_delay : 0; @@ -4912,8 +4929,7 @@ void scr_fs_rm_opts_retry(ScrStr *path, bool recursive, bool force, double max_r nanosleep(&ts, NULL); } } - scr_fs_throw(f.err, f.op, f.path); - scr_str_release(f.path); + scr_rm_fail_throw(&f); } #if defined(_WIN32) || defined(__wasi__) diff --git a/packages/runtime/src/scr_lib.h b/packages/runtime/src/scr_lib.h index bae678292..1b859cea1 100644 --- a/packages/runtime/src/scr_lib.h +++ b/packages/runtime/src/scr_lib.h @@ -306,6 +306,7 @@ void scr_fs_rm_opts(ScrStr *path, bool recursive, bool force); * EPERM failures up to maxRetries times with Node's linear backoff * (retryDelay ms longer per try); everything else throws immediately. */ void scr_fs_rm_opts_retry(ScrStr *path, bool recursive, bool force, double max_retries, double retry_delay); +bool scr_fs_rm_retry_options_ok(double max_retries, double retry_delay); ScrStr *scr_fs_mkdtemp(ScrStr *prefix); ScrStr *scr_fs_mkdtemp_async(ScrStr *prefix); /* the callback and promise error path */ void scr_fs_access(ScrStr *path, double mode); @@ -493,6 +494,9 @@ ScrPromise *scr_fsp_utimes(const ScrDyn *path, const ScrDyn *atime, const ScrDyn ScrPromise *scr_fsp_lutimes(const ScrDyn *path, const ScrDyn *atime, const ScrDyn *mtime); ScrPromise *scr_fsp_readdir(ScrStr *path); ScrPromise *scr_fsp_rm(ScrStr *path); +ScrPromise *scr_fsp_rm_opts(ScrStr *path, bool recursive, bool force); +ScrPromise *scr_fsp_rm_opts_retry(ScrStr *path, bool recursive, bool force, double max_retries, + double retry_delay); ScrPromise *scr_fsp_stat(ScrStr *path); ScrPromise *scr_fsp_realpath(ScrStr *path); ScrPromise *scr_fsp_lstat(ScrStr *path); diff --git a/packages/runtime/test/lib.test.ts b/packages/runtime/test/lib.test.ts index fbecaff37..2a32202b7 100644 --- a/packages/runtime/test/lib.test.ts +++ b/packages/runtime/test/lib.test.ts @@ -62,7 +62,8 @@ afterAll(async () => { // fs failure through scr_exc_print_uncaught. fs failures are Error // INSTANCES whose messages carry Node's exact shape (": , // ''"), so the uncaught line renders "Error: " — -// the same first line Node prints for an uncaught fs error. Typed catches +// the same first line Node prints for an uncaught fs error (rm on a +// directory is Node's ERR_FS_EISDIR SystemError instead). Typed catches // observe the message itself (`e.message`, corpus-covered differentially); // these lines pin OUR uncaught rendering. test("process + fs library: checks pass, error messages match Node's shape", async () => { @@ -76,7 +77,7 @@ test("process + fs library: checks pass, error messages match Node's shape", asy `Uncaught Error: ENOENT: no such file or directory, scandir '${scratch}/missing.txt'`, `Uncaught Error: EEXIST: file already exists, mkdir '${scratch}/sub'`, `Uncaught Error: ENOENT: no such file or directory, mkdir '${scratch}/nope/deep'`, - `Uncaught Error: EISDIR: illegal operation on a directory, rm '${scratch}/sub'`, + `Uncaught SystemError [ERR_FS_EISDIR]: Path is a directory: rm returned EISDIR (is a directory) ${scratch}/sub`, ``, ].join("\n"), ); diff --git a/tests/corpus/6930-fs-promises-rm-options.ts b/tests/corpus/6930-fs-promises-rm-options.ts new file mode 100644 index 000000000..e6de67fdd --- /dev/null +++ b/tests/corpus/6930-fs-promises-rm-options.ts @@ -0,0 +1,99 @@ +// fs.promises.rm with an options object: { recursive, force } and the +// maxRetries/retryDelay pair, as rmSync accepts them. Failures reject with +// Node's errors: a missing path without force, a directory without +// recursive (ERR_FS_EISDIR), and an out-of-range maxRetries or retryDelay. +// rmSync throws the same errors, checking its options before the path, +// while fs.promises.rm checks the path first. +import * as fs from "node:fs"; +import * as fsp from "node:fs/promises"; +import * as os from "node:os"; +import * as path from "node:path"; + +const dir = fs.mkdtempSync(path.join(os.tmpdir(), "fsp-rm-")); + +function report(label: string, error: unknown): void { + if (error instanceof Error) { + const code = (error as NodeJS.ErrnoException).code; + console.log(label, error.name, code, error.message.split(dir).join("")); + } else { + console.log(label, "non-error"); + } +} + +async function settle(label: string, run: () => Promise): Promise { + try { + await run(); + console.log(label, "fulfilled"); + } catch (error) { + report(label, error); + } +} + +function attempt(label: string, run: () => void): void { + try { + run(); + console.log(label, "returned"); + } catch (error) { + report(label, error); + } +} + +function tree(root: string): void { + fs.mkdirSync(path.join(root, "a", "b"), { recursive: true }); + fs.writeFileSync(path.join(root, "a", "b", "leaf.txt"), "leaf"); + fs.writeFileSync(path.join(root, "top.txt"), "top"); +} + +async function main(): Promise { + const missing = path.join(dir, "missing"); + await settle("missing force", () => fsp.rm(missing, { recursive: true, force: true })); + await settle("missing force only", () => fsp.rm(missing, { force: true })); + await settle("missing", () => fsp.rm(missing, { recursive: true })); + await settle("missing empty options", () => fsp.rm(missing, {})); + + const file = path.join(dir, "file.txt"); + fs.writeFileSync(file, "x"); + await settle("file", () => fsp.rm(file, { force: false })); + console.log("file exists", fs.existsSync(file)); + + const nested = path.join(dir, "nested"); + tree(nested); + await settle("dir without recursive", () => fsp.rm(nested, { force: true })); + console.log("dir exists", fs.existsSync(nested)); + await settle("dir recursive", () => fsp.rm(nested, { recursive: true })); + console.log("dir exists", fs.existsSync(nested)); + tree(nested); + await settle("fs.promises.rm", () => fs.promises.rm(nested, { recursive: true, force: true })); + console.log("dir exists", fs.existsSync(nested)); + + const retried = path.join(dir, "retried"); + tree(retried); + await settle("retry", () => fsp.rm(retried, { recursive: true, force: true, maxRetries: 2, retryDelay: 10 })); + console.log("retried exists", fs.existsSync(retried)); + await settle("retry missing", () => fsp.rm(retried, { maxRetries: 1 })); + await settle("negative retries", () => fsp.rm(retried, { force: true, maxRetries: -1 })); + await settle("fractional delay", () => fsp.rm(retried, { force: true, retryDelay: 1.5 })); + + await settle("null byte and range", () => fsp.rm("a\0b", { maxRetries: -1 })); + await settle("too many retries", () => fsp.rm(retried, { force: true, maxRetries: 2 ** 32 })); + await settle("long delay", () => fsp.rm(retried, { force: true, retryDelay: 2 ** 31 })); + await settle("delay before retries", () => fsp.rm(retried, { maxRetries: -1, retryDelay: 0.5 })); + + tree(nested); + attempt("rmSync dir", () => fs.rmSync(nested)); + attempt("rmSync dir force", () => fs.rmSync(nested, { force: true })); + attempt("rmSync null byte and range", () => fs.rmSync("a\0b", { maxRetries: -1 })); + attempt("rmSync negative delay", () => fs.rmSync(nested, { recursive: true, retryDelay: -1 })); + attempt("rmSync retry", () => fs.rmSync(nested, { recursive: true, maxRetries: 1 })); + console.log("dir exists", fs.existsSync(nested)); + + // The promise is created before the await, and a rejection is caught + // through .catch like any other promise. + const pending = fsp.rm(missing, { recursive: false }); + await pending.catch((error: unknown) => report("caught", error)); + + fs.rmSync(dir, { recursive: true, force: true }); + console.log("done", fs.existsSync(dir)); +} + +void main(); diff --git a/tests/harness/corpus-costs/6930-fs-promises-rm-options.ts.json b/tests/harness/corpus-costs/6930-fs-promises-rm-options.ts.json new file mode 100644 index 000000000..738c081aa --- /dev/null +++ b/tests/harness/corpus-costs/6930-fs-promises-rm-options.ts.json @@ -0,0 +1 @@ +{ "plain": 1965, "san": 46298 } From d78bd871f68f7e038c056cb71298d8a73c5de1a1 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 09:16:06 -0500 Subject: [PATCH 22/51] Clear recycled blocks for compact instances and keep TypeScript 7 parity --- packages/compiler/src/backend/llvm/alloc.ts | 7 ++- .../compiler/src/backend/llvm/classes.test.ts | 25 ++++++++ packages/compiler/src/backend/llvm/classes.ts | 4 +- ...-contract-program-lifetime-layouts.ts.json | 6 ++ .../4635-contract-program-lifetime-layouts.ts | 61 +++++++++++++++++++ ...-contract-program-lifetime-layouts.ts.json | 2 +- 6 files changed, 100 insertions(+), 5 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/4635-contract-program-lifetime-layouts.ts.json diff --git a/packages/compiler/src/backend/llvm/alloc.ts b/packages/compiler/src/backend/llvm/alloc.ts index c1fc5998d..c28a63e62 100644 --- a/packages/compiler/src/backend/llvm/alloc.ts +++ b/packages/compiler/src/backend/llvm/alloc.ts @@ -269,8 +269,9 @@ export function cycTypeDescriptor(name: string, trace: string, free: string, kin * expression) in `%o`, aborting on OOM, with the RC-audit allocation note * on the out-of-line path. With `cyc`, the object carries a collector * header (scr_cyc_alloc's contract). `stored` is the number of leading - * object bytes the caller stores unconditionally right after (at least the - * refcount word), which a recycled block need not clear. Labels use the + * object bytes the caller stores unconditionally right after (the refcount + * word, or the 4-byte header of a compact class instance), which a recycled + * block need not clear; it is a multiple of 4 and at most `size`. Labels use the * `sa.` prefix; the lines end in an open block. */ export function emitObjectAlloc( @@ -350,7 +351,7 @@ function allocLines( ` store ptr %sa.next, ptr %sa.fp`, ` %sa.z = getelementptr inbounds i8, ptr %sa.b, i64 ${keep}`, ` %sa.zn = sub i64 %sa.n, ${keep}`, - ` call void @llvm.memset.p0.i64(ptr align 8 %sa.z, i8 0, i64 %sa.zn, i1 false)`, + ` call void @llvm.memset.p0.i64(ptr align ${keep % 8 === 0 ? 8 : 4} %sa.z, i8 0, i64 %sa.zn, i1 false)`, ` br label %sa.fast`, `sa.bump: ; fresh reservation memory is already zero`, ` %sa.c1 = add i64 %sa.c, 1`, diff --git a/packages/compiler/src/backend/llvm/classes.test.ts b/packages/compiler/src/backend/llvm/classes.test.ts index 3c2b0095c..2f554b458 100644 --- a/packages/compiler/src/backend/llvm/classes.test.ts +++ b/packages/compiler/src/backend/llvm/classes.test.ts @@ -189,6 +189,31 @@ test("ProgramLifetime classes pack their fields after a 32-bit header", () => { expect(mortal).toContain(`%${mangleClassStruct("Leaf")} = type { i64, ptr, i8, i32, ptr, i8 }`); }); +test("a recycled block is cleared from the end of a ProgramLifetime instance's 32-bit header", () => { + const BOOL = { kind: "bool" as const }; + const mod: IrModule = { + irVersion: 17, + sourceFile: loc.file, + entry: "main", + functions: [{ name: "main", params: [], returnType: VOID, locals: [], body: [], loc }], + classes: [ + { name: "Marker", fields: [], methods: [], loc }, + { name: "Toggle", fields: [{ name: "on", type: BOOL }], methods: [], loc }, + ], + contracts: { programLifetime: ["Marker", "Toggle"] }, + }; + const llvm = emitLlvmModule(mod, { pointerBits: 64 }); + for (const name of ["Marker", "Toggle"]) { + const start = llvm.indexOf(`define internal ptr @${mangleClassNew(name)}(`); + const body = llvm.slice(start, llvm.indexOf("\n}\n", start)); + // The constructor stores the 4-byte header itself; a 4-byte instance + // (Marker) clears nothing, and Toggle's field at offset 4 is cleared. + expect(body).toContain("%sa.z = getelementptr inbounds i8, ptr %sa.b, i64 4"); + expect(body).toContain("%sa.zn = sub i64 %sa.n, 4"); + expect(body).toContain("call void @llvm.memset.p0.i64(ptr align 4 %sa.z,"); + } +}); + test("published ProgramLifetime instances use the published mark every other object uses", () => { const node = { kind: "object" as const, className: "Node" }; const leaf = { kind: "object" as const, className: "Leaf" }; diff --git a/packages/compiler/src/backend/llvm/classes.ts b/packages/compiler/src/backend/llvm/classes.ts index 061bd06a2..1634f1f9e 100644 --- a/packages/compiler/src/backend/llvm/classes.ts +++ b/packages/compiler/src/backend/llvm/classes.ts @@ -1098,7 +1098,9 @@ export function emitClassShapes( host, sizeOf, traced && !lifetime ? { type: cycType } : null, - meta.hierarchy && !meta.idInRc ? 16 : 8, + // The words stored next: a compact instance's 4-byte header, the + // rc word, or the rc word and the vtable word. + meta.compact ? 4 : meta.hierarchy && !meta.idInRc ? 16 : 8, ), ); if (meta.compact && !meta.hierarchy) diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/4635-contract-program-lifetime-layouts.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/4635-contract-program-lifetime-layouts.ts.json new file mode 100644 index 000000000..31a42c38b --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/4635-contract-program-lifetime-layouts.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/4635-contract-program-lifetime-layouts.ts" + ], + "diags": [] +} diff --git a/tests/corpus/4635-contract-program-lifetime-layouts.ts b/tests/corpus/4635-contract-program-lifetime-layouts.ts index b62b7fa1a..7b300091e 100644 --- a/tests/corpus/4635-contract-program-lifetime-layouts.ts +++ b/tests/corpus/4635-contract-program-lifetime-layouts.ts @@ -222,3 +222,64 @@ const fresh = new Binary(new Ident(16, "y"), "-", new Literal(20, 3)); fresh.precedence = 11; fresh.right.flagged = true; console.log("fresh", describe(fresh), describe(fresh.right), classify(fresh.left), classify(published.root)); + +// Instances in recycled blocks: freed 8- and 16-byte objects leave their +// blocks on the allocator's free lists, and the next instances of the same +// sizes reuse them. Marker has no fields (a 4-byte instance); Toggle's +// field and Pair's first field sit right after the header. +class Scrap { + readonly n: number; + constructor(n: number) { + this.n = n; + } +} + +class Blank {} + +class Marker implements ProgramLifetime {} + +class Toggle implements ProgramLifetime { + on: boolean; + constructor(on: boolean) { + this.on = on; + } +} + +class Pair implements ProgramLifetime { + readonly low: number; + readonly high: Node | undefined; + constructor(low: number, high: Node | undefined) { + this.low = low | 0; + this.high = high; + } +} + +const blanks: Blank[] = []; +const scraps: Scrap[] = []; +const markers: Marker[] = []; +const toggles: Toggle[] = []; +const pairs: Pair[] = []; +for (let round = 0; round < 3; round++) { + for (let i = 0; i < 200; i++) { + blanks.push(new Blank()); + scraps.push(new Scrap(i * 1.5)); + } + let sum = 0; + for (const s of scraps) sum += s.n; + blanks.length = 0; + scraps.length = 0; + for (let i = 0; i < 200; i++) { + markers.push(new Marker()); + toggles.push(new Toggle(i % 3 === 0)); + pairs.push(new Pair(i - 100, i % 2 === 0 ? undefined : published.root)); + } + let on = 0; + let low = 0; + let high = 0; + for (const t of toggles) if (t.on) on++; + for (const p of pairs) { + low += p.low; + if (p.high !== undefined) high++; + } + console.log("recycled", round, sum, markers.length, on, low, high); +} diff --git a/tests/harness/corpus-costs/4635-contract-program-lifetime-layouts.ts.json b/tests/harness/corpus-costs/4635-contract-program-lifetime-layouts.ts.json index 2b790bed2..4eba5d094 100644 --- a/tests/harness/corpus-costs/4635-contract-program-lifetime-layouts.ts.json +++ b/tests/harness/corpus-costs/4635-contract-program-lifetime-layouts.ts.json @@ -1 +1 @@ -{ "plain": 4944, "san": 7176 } +{ "plain": 4944, "san": 7757 } From 9d898d0affc1a32146c41962ec219a0aec4dea85 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 09:25:11 -0500 Subject: [PATCH 23/51] Name the stored header size for compact instances --- packages/compiler/src/backend/llvm/classes.ts | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/packages/compiler/src/backend/llvm/classes.ts b/packages/compiler/src/backend/llvm/classes.ts index 1634f1f9e..9963fad5b 100644 --- a/packages/compiler/src/backend/llvm/classes.ts +++ b/packages/compiler/src/backend/llvm/classes.ts @@ -569,6 +569,16 @@ export const COMPACT_HEADER_TAG = 0x70000000; export const COMPACT_PUBLISHED = 0x80000000; export const COMPACT_ID_MASK = 0x0fffffff; +/** The leading bytes of a new instance that its allocation function stores + * right after allocating it (emitObjectAlloc's `stored`), which a recycled + * block therefore need not clear: a compact instance's 4-byte header word + * (its fields start at offset 4), the 8-byte rc word, or the rc word and + * the vtable word of a hierarchy member that keeps its class in a vtable. */ +export function storedHeaderBytes(meta: LlClassMeta): number { + if (meta.compact) return 4; + return meta.hierarchy && !meta.idInRc ? 16 : 8; +} + /** The header word a compact instance is created with. */ export function compactHeaderWord(meta: LlClassMeta): number { return COMPACT_HEADER_TAG | (meta.hierarchy ? meta.pre + 1 : 0); @@ -1098,9 +1108,7 @@ export function emitClassShapes( host, sizeOf, traced && !lifetime ? { type: cycType } : null, - // The words stored next: a compact instance's 4-byte header, the - // rc word, or the rc word and the vtable word. - meta.compact ? 4 : meta.hierarchy && !meta.idInRc ? 16 : 8, + storedHeaderBytes(meta), ), ); if (meta.compact && !meta.hierarchy) From 5a56cc2564e99686c71a5cdd10e0da2d8fcdaf16 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 09:40:20 -0500 Subject: [PATCH 24/51] Answer catch-binding instanceof for values thrown as unknown A union, unknown or any value is converted to a dynamic value before it is thrown, and promise rejections and other runtime throws carry dynamic values too. A catch binding then holds a dynamic payload, which the native class test does not recognize: `e instanceof C` was false for every class of a hierarchy, and a read narrowed by it trapped. A test through a parameter of type unknown was already correct. When a class or one of its subclasses crosses into unknown, the class dynamic-dispatch pass now routes the catch-binding tests of that class (`e instanceof C`, the reads it narrows and `e as C`) through helpers that keep the native test and answer a dynamic payload with the dynamic class test, as standalone classes already do. The new "dynamic" caught test selects that payload, so native payloads are not converted. Programs whose classes never cross into unknown compile as before. --- .../compiler/src/backend/llvm/expr-dynamic.ts | 23 +++ .../lowering/caught-value-dispatch.ts | 176 +++++++++++++++++- .../lowering/class-dynamic-dispatch.ts | 9 + packages/compiler/src/ir/ir.ts | 6 +- packages/compiler/src/ir/validate.ts | 2 +- ...caught-instanceof-dynamic-payloads.ts.json | 6 + ...6819-caught-instanceof-dynamic-payloads.ts | 80 ++++++++ ...caught-instanceof-dynamic-payloads.ts.json | 1 + 8 files changed, 299 insertions(+), 4 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6819-caught-instanceof-dynamic-payloads.ts.json create mode 100644 tests/corpus/6819-caught-instanceof-dynamic-payloads.ts create mode 100644 tests/harness/corpus-costs/6819-caught-instanceof-dynamic-payloads.ts.json diff --git a/packages/compiler/src/backend/llvm/expr-dynamic.ts b/packages/compiler/src/backend/llvm/expr-dynamic.ts index 83cbe6d54..5b5ec33bc 100644 --- a/packages/compiler/src/backend/llvm/expr-dynamic.ts +++ b/packages/compiler/src/backend/llvm/expr-dynamic.ts @@ -1038,6 +1038,29 @@ export function emitDynamicExpr( B.line(`${n} = xor i1 ${t}, true`); return { name: n, type: e.type }; } + if (e.test === "dynamic") { + // A REF payload whose retain adapter is the dyn adapter (sc_cd's + // by-reference arm). + host.declare(`declare ptr @scr_dyn_retain_v(ptr)`); + const kp = B.tmp(), + k = B.tmp(), + isRef = B.tmp(), + rp = B.tmp(), + r = B.tmp(), + isDyn = B.tmp(), + t = B.tmp(); + B.line(`${kp} = getelementptr inbounds %ScrCaught, ptr ${c.name}, i64 0, i32 1`); + B.line(`${k} = load i32, ptr ${kp}`); + B.line(`${isRef} = icmp eq i32 ${k}, 4`); + B.line(`${rp} = getelementptr inbounds %ScrCaught, ptr ${c.name}, i64 0, i32 5`); + B.line(`${r} = load ptr, ptr ${rp}`); + B.line(`${isDyn} = icmp eq ptr ${r}, @scr_dyn_retain_v`); + B.line(`${t} = and i1 ${isRef}, ${isDyn}`); + if (e.negated !== true) return { name: t, type: e.type }; + const n = B.tmp(); + B.line(`${n} = xor i1 ${t}, true`); + return { name: n, type: e.type }; + } if (e.test === "object") { host.declare(`declare zeroext i1 @scr_caught_is_object(ptr)`); const t = B.tmp(); diff --git a/packages/compiler/src/frontend/lowering/caught-value-dispatch.ts b/packages/compiler/src/frontend/lowering/caught-value-dispatch.ts index a07741461..645ff8d2c 100644 --- a/packages/compiler/src/frontend/lowering/caught-value-dispatch.ts +++ b/packages/compiler/src/frontend/lowering/caught-value-dispatch.ts @@ -1,6 +1,16 @@ import { varRef } from "../../ir/build.js"; -import { BOOL, CAUGHT, DYN, type IrFunction, type IrType } from "../../ir/ir.js"; +import { + BOOL, + CAUGHT, + DYN, + type IrExpr, + type IrFunction, + type IrType, + type SrcLoc, +} from "../../ir/ir.js"; import { everyStmtList, transformStmtList } from "../../ir/traverse.js"; +import { classInstanceOf } from "./class-dynamic-dispatch.js"; +import type { ClassInfo } from "./lower-classes.js"; import type { Lowerer } from "./lowerer.js"; /** Exception snapshots retain native objects. Recover their known layouts @@ -10,6 +20,10 @@ export class CaughtValueDispatch { private readonly thrownClasses = new Set(); private readonly convertedClasses = new Set(); private helper: IrFunction | null = null; + /** The dynamic-payload helpers of class tests and extractions, by kind + * and class (dynamicPayloads). */ + private readonly payloadHelpers = new Map(); + private readonly payloadHelperFns = new Set(); process(lowerer: Lowerer, functions: readonly IrFunction[]): boolean { const consumers: IrFunction[] = []; @@ -81,4 +95,164 @@ export class CaughtValueDispatch { } return changed; } + + /** A thrown unknown or union value, and the runtime's own dynamic throws, + * reach a catch binding as a dynamic payload. When an instance of class C + * can be such a payload (C or a subclass of C crosses into unknown, the + * `boxed` classes), `e instanceof C`, the reads it narrows and `e as C` in + * `sites` also answer through the dynamic class test, as standalone classes + * already do. Programs that never box C keep the native payload test. */ + dynamicPayloads( + lowerer: Lowerer, + sites: ReadonlySet, + boxed: ReadonlySet, + ): boolean { + if (boxed.size === 0 || sites.size === 0) return false; + const reached = new Map(); + const reachable = (className: string | undefined): ClassInfo | null => { + if (className === undefined) return null; + let info = reached.get(className); + if (info === undefined) { + const found = lowerer.classes.get(className); + info = + found && + lowerer.inHierarchy(found) && + [...boxed].some((name) => name === className || lowerer.isSubclassOf(name, className)) + ? found + : null; + reached.set(className, info); + } + return info; + }; + let changed = false; + const qualifies = (expr: IrExpr): boolean => + expr.kind === "caughtTest" + ? expr.test === "instanceof" && reachable(expr.className) !== null + : (expr.kind === "caughtNarrow" || expr.kind === "caughtCheck") && + expr.type.kind === "object" && + reachable(expr.type.className) !== null; + for (const fn of sites) { + // The conversions themselves test the native payload. + if (fn === this.helper || this.payloadHelperFns.has(fn)) continue; + const found = !everyStmtList(fn.body, { stmt: () => true, expr: (expr) => !qualifies(expr) }); + if (!found) continue; + fn.body = transformStmtList(fn.body, { + stmt: (stmt) => stmt, + expr: (expr) => { + if (expr.kind === "caughtTest" && expr.test === "instanceof") { + const info = reachable(expr.className); + if (!info) return expr; + changed = true; + const call: IrExpr = { + kind: "call", + callee: this.payloadHelper(lowerer, "instanceof", info, expr.loc).name, + args: [expr.value], + type: BOOL, + loc: expr.loc, + }; + return expr.negated + ? { kind: "unary", op: "!", operand: call, type: BOOL, loc: expr.loc } + : call; + } + if ( + (expr.kind === "caughtNarrow" || expr.kind === "caughtCheck") && + expr.type.kind === "object" + ) { + const info = reachable(expr.type.className); + if (!info) return expr; + changed = true; + const kind = expr.kind === "caughtNarrow" ? "narrow" : "check"; + return { + kind: "call", + callee: this.payloadHelper(lowerer, kind, info, expr.loc).name, + args: [expr.value], + type: expr.type, + loc: expr.loc, + }; + } + return expr; + }, + }); + } + return changed; + } + + /** `%caught.instanceof:C`, `%caught.narrow:C` and `%caught.check:C`: the + * native payload test or extraction, and otherwise a value thrown as + * unknown tested against C and checked into C. A check of a payload that + * is not a C keeps caughtCheck's TypeError. */ + private payloadHelper( + lowerer: Lowerer, + kind: "instanceof" | "narrow" | "check", + info: ClassInfo, + loc: SrcLoc, + ): IrFunction { + const className = info.def.name; + const name = `%caught.${kind}:${className}`; + const existing = this.payloadHelpers.get(name); + if (existing) return existing; + const caught = (): IrExpr => varRef("caught", CAUGHT, loc); + const dynamic = (): IrExpr => ({ kind: "caughtToDyn", value: caught(), type: DYN, loc }); + const native: IrExpr = { + kind: "caughtTest", + value: caught(), + test: "instanceof", + className, + type: BOOL, + loc, + }; + // Only a value thrown as unknown can hold an instance besides a native + // payload, which the native test answers exactly. + const dynamicInstance = (): IrExpr => ({ + kind: "logical", + op: "&&", + left: { kind: "caughtTest", value: caught(), test: "dynamic", type: BOOL, loc }, + right: classInstanceOf(lowerer, dynamic(), info, loc), + type: BOOL, + loc, + }); + const type: IrType = { kind: "object", className }; + const returnType = kind === "instanceof" ? BOOL : type; + let value: IrExpr; + if (kind === "instanceof") { + value = { + kind: "ternary", + cond: native, + then: { kind: "boolLit", value: true, type: BOOL, loc }, + else_: dynamicInstance(), + type: BOOL, + loc, + }; + } else if (kind === "narrow") { + value = { + kind: "ternary", + cond: native, + then: { kind: "caughtNarrow", value: caught(), type, loc }, + else_: lowerer.coerceToExpected(dynamic(), type), + type, + loc, + }; + } else { + value = { + kind: "ternary", + cond: dynamicInstance(), + then: lowerer.coerceToExpected(dynamic(), type), + else_: { kind: "caughtCheck", value: caught(), className, type, loc }, + type, + loc, + }; + } + const helper: IrFunction = { + name, + params: [{ localId: "caught", name: "caught", type: CAUGHT }], + locals: [{ id: "caught", name: "caught", type: CAUGHT, mutable: false }], + returnType, + body: [{ kind: "return", value, loc }], + loc, + }; + this.payloadHelpers.set(name, helper); + this.payloadHelperFns.add(helper); + lowerer.liftedFns.push(helper); + return helper; + } } diff --git a/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts b/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts index cb5c28444..252e3fa4b 100644 --- a/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts +++ b/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts @@ -762,6 +762,7 @@ export class ClassDynamicDispatch { const rewrite = new Set(); const toPrimitiveSites = new Set(); const hasKeySites = new Set(); + const caughtSites = new Set(); const discover = (type: IrType): void => { const key = typeKey(type); if (seenTypes.has(key)) return; @@ -827,6 +828,13 @@ export class ClassDynamicDispatch { case "dynHasKey": hasKeySites.add(fn); break; + case "caughtTest": + if (expr.test === "instanceof") caughtSites.add(fn); + break; + case "caughtNarrow": + case "caughtCheck": + if (expr.type.kind === "object") caughtSites.add(fn); + break; case "dynKeyGet": case "dynInvoke": rewrite.add(fn); @@ -865,6 +873,7 @@ export class ClassDynamicDispatch { return true; }, }); + changed = this.caughtDispatch.dynamicPayloads(lowerer, caughtSites, this.boxed) || changed; if (this.boxed.size === 0 && rewrite.size === 0) return changed; const converting = this.convertingHints(lowerer); const calling = this.calledConversions(lowerer); diff --git a/packages/compiler/src/ir/ir.ts b/packages/compiler/src/ir/ir.ts index 359d45ee7..38ebe4561 100644 --- a/packages/compiler/src/ir/ir.ts +++ b/packages/compiler/src/ir/ir.ts @@ -3544,11 +3544,13 @@ export type IrExpr = * excluding callable and primitive references; * "instanceof" requires `className`: hierarchy payloads use the vtable * interval, while standalone classes use their exact retain adapter. - * Other payloads answer false. `negated` flips the result (the `!==` spelling). */ + * Other payloads answer false. "dynamic" is a value thrown as unknown + * (the payload caughtToDyn returns by reference). `negated` flips the + * result (the `!==` spelling). */ | { kind: "caughtTest"; value: IrExpr; - test: "string" | "number" | "boolean" | "object" | "instanceof"; + test: "string" | "number" | "boolean" | "object" | "instanceof" | "dynamic"; className?: string; negated?: boolean; type: IrType; diff --git a/packages/compiler/src/ir/validate.ts b/packages/compiler/src/ir/validate.ts index 9635ceb91..74f526e06 100644 --- a/packages/compiler/src/ir/validate.ts +++ b/packages/compiler/src/ir/validate.ts @@ -1117,7 +1117,7 @@ function validateFunction( // helpers. They are not source-level types and never return as values. if (fn.returnType.kind === "caught") err("return type is caught", fn.loc); for (const p of fn.params) { - if (p.type.kind === "caught" && fn.name !== "%caught.dynamicValue") + if (p.type.kind === "caught" && !fn.name.startsWith("%caught.")) err(`param "${p.name}" is caught-typed`, fn.loc); if (!locals.has(p.localId)) { err(`param "${p.name}" has no local entry "${p.localId}"`, fn.loc); diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6819-caught-instanceof-dynamic-payloads.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6819-caught-instanceof-dynamic-payloads.ts.json new file mode 100644 index 000000000..6e7228d5a --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6819-caught-instanceof-dynamic-payloads.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6819-caught-instanceof-dynamic-payloads.ts" + ], + "diags": [] +} diff --git a/tests/corpus/6819-caught-instanceof-dynamic-payloads.ts b/tests/corpus/6819-caught-instanceof-dynamic-payloads.ts new file mode 100644 index 000000000..c42a803f5 --- /dev/null +++ b/tests/corpus/6819-caught-instanceof-dynamic-payloads.ts @@ -0,0 +1,80 @@ +// `e instanceof C` on a catch binding when the thrown value crossed into +// unknown first: a union, an unknown or any value, a rethrown binding and an +// awaited rejection. The narrowed reads and `e as C` read the same instance. +abstract class Shape { + sides = 0; +} +class Polygon extends Shape { + name = "polygon"; +} +class Square extends Polygon { + size = 3; +} +class Plain { + w = 4; +} +class CodedError extends Error { + code = 7; +} + +function report(label: string, e: unknown): void { + console.log(label, "via unknown", e instanceof Shape, e instanceof Polygon, e instanceof Square, e instanceof Plain); +} + +function attempt(label: string, body: () => void): void { + try { + body(); + } catch (e) { + report(label, e); + console.log(label, "binding", e instanceof Shape, e instanceof Polygon, e instanceof Square, e instanceof Plain, e instanceof Error, e instanceof CodedError); + console.log(label, "negated", !(e instanceof Shape), !(e instanceof Error)); + if (e instanceof Polygon) console.log(label, "narrowed", e.name, e.sides); + if (e instanceof Square) console.log(label, "square", e.size); + if (e instanceof CodedError) console.log(label, "coded", e.code, e.message); + if (e instanceof Shape && !(e instanceof Square)) console.log(label, "shape only", (e as Polygon).name); + } +} + +function throwUnknown(value: unknown): never { + throw value; +} +const choose = (n: number): Polygon | Plain | string => (n === 0 ? new Square() : n === 1 ? new Plain() : n === 2 ? new Polygon() : "text"); +const maybe = (n: number): Square | undefined => (n === 0 ? new Square() : undefined); +const failure = (n: number): CodedError | string => (n === 0 ? new CodedError("failed") : "failed"); + +for (let n = 0; n < 4; n++) attempt(`union ${n}`, () => { throw choose(n); }); +attempt("optional", () => { throw maybe(0); }); +attempt("optional undefined", () => { throw maybe(1); }); +attempt("error union", () => { throw failure(0); }); +attempt("error union string", () => { throw failure(1); }); +attempt("unknown", () => throwUnknown(new Square())); +attempt("unknown plain", () => throwUnknown(new Plain())); +attempt("unknown error", () => throwUnknown(new CodedError("unknown"))); +const anyValue: any = new Polygon(); +attempt("any", () => { throw anyValue; }); +attempt("typed", () => { const shape: Shape = new Square(); throw shape; }); +attempt("rethrow", () => { + try { + throw choose(0); + } catch (inner) { + throw inner; + } +}); + +async function main(): Promise { + try { + await Promise.reject(choose(0)); + } catch (e) { + console.log("rejected", e instanceof Shape, e instanceof Polygon, e instanceof Square, e instanceof Plain); + if (e instanceof Square) console.log("rejected square", e.size, e.name); + } + try { + await (async () => { + throw failure(0); + })(); + } catch (e) { + console.log("async error", e instanceof Error, e instanceof CodedError); + if (e instanceof CodedError) console.log("async coded", e.code); + } +} +void main(); diff --git a/tests/harness/corpus-costs/6819-caught-instanceof-dynamic-payloads.ts.json b/tests/harness/corpus-costs/6819-caught-instanceof-dynamic-payloads.ts.json new file mode 100644 index 000000000..2809fc670 --- /dev/null +++ b/tests/harness/corpus-costs/6819-caught-instanceof-dynamic-payloads.ts.json @@ -0,0 +1 @@ +{ "plain": 3200, "san": 30100 } From 77aab9c0fd76bd46f6efdb89515d5daace1773b6 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 09:41:55 -0500 Subject: [PATCH 25/51] Address review comments on prototype toString conversions --- packages/compiler/src/backend/llvm/dyn.ts | 30 +++++++++++++++---- packages/runtime/src/scr_dyn_invoke.c | 9 ++++-- packages/runtime/src/scr_json.c | 23 +++++++++++--- .../corpus/prototype-tostring-conversions.cjs | 26 ++++++++++++++++ 4 files changed, 77 insertions(+), 11 deletions(-) diff --git a/packages/compiler/src/backend/llvm/dyn.ts b/packages/compiler/src/backend/llvm/dyn.ts index 7109fae20..ff975a800 100644 --- a/packages/compiler/src/backend/llvm/dyn.ts +++ b/packages/compiler/src/backend/llvm/dyn.ts @@ -3349,9 +3349,9 @@ export class LlDyn { B.startBlock(labels.get(DYN_KIND.ARR)!); { // Array.prototype.toString: join(",") — null/undefined ELEMENTS - // print empty (unlike top level), nested arrays flatten. + // print empty (unlike top level), nested arrays flatten. The length + // is read once, as join reads it. const n = this.lenOf(B, "%d"); - const items = this.itemsOf(B, "%d"); this.i64Loop(B, "ds.ar", n, (i, brNext) => { const nz = B.tmp(); B.line(`${nz} = icmp ugt ${this.S} ${i}, 0`); @@ -3362,7 +3362,18 @@ export class LlDyn { B.line(`call void @scr_jb_putc(ptr %b, i8 44)`); B.br(lel); B.startBlock(lel); - const e = this.itemAt(B, items, i); + // An element's toString can append to, shrink or reallocate the + // array: each index reads the current items, an index past the + // current length prints empty, and the element stays retained + // while it converts. + const now = this.lenOf(B, "%d"); + const inside = B.tmp(); + B.line(`${inside} = icmp ult ${this.S} ${i}, ${now}`); + const lIn = B.newLabel("ds.in"); + const lSkip = B.newLabel("ds.sk"); + B.condBr(inside, lIn, lSkip); + B.startBlock(lIn); + const e = this.itemAt(B, this.itemsOf(B, "%d"), i); const ek = this.kindOf(B, e); const isU = B.tmp(); const isN = B.tmp(); @@ -3370,13 +3381,22 @@ export class LlDyn { B.line(`${isU} = icmp eq i32 ${ek}, ${DYN_KIND.UNDEF}`); B.line(`${isN} = icmp eq i32 ${ek}, ${DYN_KIND.NULL}`); B.line(`${unit} = or i1 ${isU}, ${isN}`); - const lSkip = B.newLabel("ds.sk"); const lRec = B.newLabel("ds.rc"); B.condBr(unit, lSkip, lRec); B.startBlock(lSkip); brNext(); B.startBlock(lRec); - B.line(`call void @sc_ds_buf(ptr %b, ptr ${e})`); + const held = this.retainDyn(B, e); + host.declare(`declare void @scr_dyn_release_v(ptr)`); + B.line(`call void @sc_ds_buf(ptr %b, ptr ${held})`); + B.line(`call void @scr_dyn_release_v(ptr ${held})`); + // The first exception an element's conversion throws ends the + // conversion; later elements are not read. + const thrown = B.tmp(); + for (const line of host.pendingTestLines(thrown)) B.line(line.trimStart()); + const lMore = B.newLabel("ds.mo"); + B.condBr(thrown, done, lMore); + B.startBlock(lMore); }); B.br(done); } diff --git a/packages/runtime/src/scr_dyn_invoke.c b/packages/runtime/src/scr_dyn_invoke.c index 92dd944ee..e413acfe9 100644 --- a/packages/runtime/src/scr_dyn_invoke.c +++ b/packages/runtime/src/scr_dyn_invoke.c @@ -72,8 +72,12 @@ static void scr_dyn_display_buf(ScrJsonBuf *b, const ScrDyn *d, bool to_string) case SCR_DYN_STR: scr_jb_write(b, d->v.str->data, d->v.str->len); return; - case SCR_DYN_ARR: - for (size_t i = 0; i < d->v.arr.len; i++) { + case SCR_DYN_ARR: { + /* The length is read once, as join reads it; an element's toString can + * append to or shrink the array (scr_dyn_arr_at answers undefined past + * its current end). */ + size_t len = d->v.arr.len; + for (size_t i = 0; i < len; i++) { if (i > 0) scr_jb_putc(b, ','); ScrDyn *e = scr_dyn_arr_at(d, (double)i); if (!e) return; @@ -82,6 +86,7 @@ static void scr_dyn_display_buf(ScrJsonBuf *b, const ScrDyn *d, bool to_string) if (scr_exc_pending()) return; } return; + } case SCR_DYN_OBJ: { const ScrDyn *marker = scr_dyn_obj_get(d, "%error", 6); if (marker) { diff --git a/packages/runtime/src/scr_json.c b/packages/runtime/src/scr_json.c index 0281e5ca6..13fc73187 100644 --- a/packages/runtime/src/scr_json.c +++ b/packages/runtime/src/scr_json.c @@ -7492,8 +7492,13 @@ ScrStr *scr_dyn_to_string(const ScrDyn *d, const ScrStr *enc) { /* Object.prototype.toString with the Promise @@toStringTag. */ return scr_str_new("[object Promise]", 16); case SCR_DYN_ARR: { + /* Array.prototype.join reads the length once and each element by index: + * an element's toString can append to or shrink the array, so an index + * past the current length reads undefined, and the element stays + * retained while it converts. */ + size_t len = d->v.arr.len; ScrStr *out = scr_str_new("", 0); - for (size_t i = 0; i < d->v.arr.len; i++) { + for (size_t i = 0; i < len; i++) { if (i > 0) { ScrStr *comma = scr_str_new(",", 1); ScrStr *joined = scr_str_concat(out, comma); @@ -7501,11 +7506,21 @@ ScrStr *scr_dyn_to_string(const ScrDyn *d, const ScrStr *enc) { scr_str_release(comma); out = joined; } - const ScrDyn *e = d->v.arr.items[i]; - if (e->kind == SCR_DYN_UNDEF || e->kind == SCR_DYN_NULL) continue; /* JS join: empty */ + ScrDyn *e = scr_dyn_arr_at(d, (double)i); + if (e->kind == SCR_DYN_UNDEF || e->kind == SCR_DYN_NULL) { /* JS join: empty */ + scr_dyn_release(e); + if (scr_exc_pending()) { + scr_str_release(out); + return NULL; + } + continue; + } /* ToString with the object protocol: an element's own or inherited * toString/valueOf runs, as in Array.prototype.join. */ - ScrStr *piece = scr_dyn_string_coerce_js(e); + ScrStr *piece = NULL; + if (e->kind == SCR_DYN_PROXY) scr_dyn_proxy_unsupported("string conversion"); + else piece = scr_dyn_string_coerce_js(e); + scr_dyn_release(e); if (!piece || scr_exc_pending()) { scr_str_release(piece); scr_str_release(out); diff --git a/tests/corpus/prototype-tostring-conversions.cjs b/tests/corpus/prototype-tostring-conversions.cjs index b4983893f..a5c6ba83c 100644 --- a/tests/corpus/prototype-tostring-conversions.cjs +++ b/tests/corpus/prototype-tostring-conversions.cjs @@ -38,3 +38,29 @@ try { console.log(`${thrower}`); } catch (e) { console.log("template", e.message try { console.log([thrower].join()); } catch (e) { console.log("join", e.message); } try { console.log(String([thrower])); } catch (e) { console.log("String", e.message); } try { console.log([new RefNum(1), thrower].sort()); } catch (e) { console.log("sort", e.message); } + +// The conversions read the array's length once: elements an element's +// toString appends are not converted, an index it removes reads empty, and +// the first exception ends the conversion before later elements convert. +const growing = []; +growing.push({ toString() { growing.push("late"); return "g"; } }, "b"); +console.log(`${growing}`, growing.length); +console.log(growing.join("|"), growing.length); +console.log(String(growing), growing.length); +const shrinking = []; +shrinking.push({ toString() { shrinking.length = 1; return "s"; } }, "gone", "also"); +console.log(`${shrinking}`, shrinking.length); +shrinking.push("gone", "also"); +console.log(shrinking.join("|"), shrinking.length); +shrinking.push("gone", "also"); +console.log(String(shrinking), shrinking.length); +const visited = []; +const stops = []; +stops.push( + { toString() { throw new Error("first"); } }, + { toString() { visited.push("second"); return "second"; } }, +); +try { console.log(`${stops}`); } catch (e) { console.log("template", e.message, visited.length); } +try { console.log(stops.join()); } catch (e) { console.log("join", e.message, visited.length); } +try { console.log(String(stops)); } catch (e) { console.log("String", e.message, visited.length); } +try { console.log(`${[stops]}`); } catch (e) { console.log("nested", e.message, visited.length); } From 3cc72ac96ca01a095a2fc6412cffdff143de9c21 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 09:51:40 -0500 Subject: [PATCH 26/51] Prepare v0.4.1 release --- CHANGELOG.md | 75 ++++++++++++++++++- packages/cli-darwin-arm64/package.json | 2 +- packages/cli-darwin-x64/package.json | 2 +- packages/cli-linux-arm64-gnu/package.json | 2 +- packages/cli-linux-arm64-musl/package.json | 2 +- packages/cli-linux-x64-gnu/package.json | 2 +- packages/cli-linux-x64-musl/package.json | 2 +- packages/cli-win32-x64-msvc/package.json | 2 +- packages/cli/package.json | 2 +- packages/compiler/package.json | 2 +- packages/compiler/surface-manifest.json | 2 +- packages/experimental/package.json | 2 +- packages/llvm-darwin-arm64/package.json | 2 +- packages/llvm-darwin-x64/package.json | 2 +- packages/llvm-linux-arm64-gnu/package.json | 2 +- packages/llvm-linux-arm64-musl/package.json | 2 +- packages/llvm-linux-x64-gnu/package.json | 2 +- packages/llvm-linux-x64-musl/package.json | 2 +- packages/llvm-win32-x64-msvc/package.json | 2 +- packages/runtime-android-arm64/package.json | 2 +- packages/runtime-darwin-arm64/package.json | 2 +- packages/runtime-darwin-x64/package.json | 2 +- packages/runtime-ios-arm64/package.json | 2 +- .../runtime-ios-simulator-arm64/package.json | 2 +- packages/runtime-linux-arm64-gnu/package.json | 2 +- .../runtime-linux-arm64-musl/package.json | 2 +- packages/runtime-linux-x64-gnu/package.json | 2 +- packages/runtime-linux-x64-musl/package.json | 2 +- packages/runtime-wasm32-wasi/package.json | 2 +- packages/runtime-win32-x64-msvc/package.json | 2 +- packages/runtime/package.json | 2 +- packages/threads/package.json | 2 +- packages/types/package.json | 2 +- 33 files changed, 105 insertions(+), 34 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c319a1b53..31d34e8db 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,79 @@ All notable changes to scriptc will be documented in this file. +## 0.4.1 + +### Highlights + +- **Profile-guided optimization:** `scriptc build --profile-generate` builds an instrumented executable, `scriptc merge-profiles` (with `--sparse`) merges the profiles its runs write, and `--profile-use` optimizes a release or speed build with the merged profile. tsc-ts built with a profile from its own type check checks other projects 1.15 to 1.2 times as fast, with the same diagnostics and memory. Builds without a profile are unchanged. +- **Faster cold builds of large programs:** large class-based programs build in partitions instead of as one module, so their cold builds are 25 to 32% faster, and an edit to one function recompiles only its partition. +- **npm packages:** more packages that pass builtins, classes and promises around as values now compile from their JavaScript. Among them are execa 5, hot-shots, @opentelemetry/api, async-sema, buffer-crc32, yazl, tar-stream, smol-toml and truncate-utf8-bytes. `undici` 5.x and 7.x compile from an implementation that ships with scriptc, and `isarray`, `safe-buffer` and `process-nextick-args` compile as the Node builtins they stand in for. +- **Dynamic-tier property access goes through per-site inline caches** keyed on object shapes. On constructor-and-prototype code, the code that reads several object shapes at one site and an AST interpreter, untyped code runs 1.4 to 3 times as fast as in 0.4.0. Typed programs compile to the same code as before. +- **Smaller arrays, class instances and number-keyed maps:** + - The array header is 48 bytes instead of 64. + - Small arrays start with three inline slots. + - Instances of program class hierarchies keep their class id in the reference-count word instead of a vtable pointer. + - `Map`s and `Set`s with number keys store 16-byte entries. + - Together they lower the peak memory of a large TypeScript type checker by 8 to 10%, with the same output and wall time. + +### Features + +- **`for await` iterates untyped values,** including async generators, `Symbol.asyncIterator` results and synchronous iterators, whose values are awaited. `yield*` delegates inside async generators. A JavaScript generator whose return type is inferred as `any` compiles, including one that delegates to itself. +- **`crypto.createECDH` and `ECDH.convertKey`** are supported for the 11 short Weierstrass curves mbedTLS implements, with Node 24's key formats, encodings and errors. Other curves are refused. +- **`undici` 5.x and 7.x:** `Client`, `Pool`, `Agent`, `request`, the global dispatcher, `fetch` through a dispatcher, the error classes and the `undici:` diagnostics channels follow each version's behavior. Each importing package compiles against its own installed version. The limitations page lists the parts that differ. +- **Packages marked as never loaded:** `"scriptc": { "packages": { "": "unloaded" } }` in the entry's `package.json` makes `require("")` throw Node's `MODULE_NOT_FOUND`, so optional dependencies take their absent branch. +- **`util.promisify` of an untyped function** returns a promise-returning wrapper at run time. +- **Destructuring with runtime computed keys** (`const { [name]: v } = source`) reads records and untyped values. A missing key gives `undefined` and the element default applies. +- **Import cycles compile when a class is used only after both modules finish initializing.** Reaching the class earlier throws Node's `ReferenceError: Cannot access 'X' before initialization`. +- **`await` of an untyped value adopts thenables,** including objects that define `then` at run time. `Promise.prototype` and `promise.constructor` can be read as values. A child process can be given new properties and awaited as a thenable. +- **Function values convert their results and arguments** when they are stored in a slot whose signature has the same structure but a different shape, such as a function returning `Promise` stored in a `() => Promise` slot. +- **Declarations from SDKs and shared type packages:** index signatures of `any`, `any[]` members, classes that a package only declares, and `Pick` views of classes compile. +- **Builtins as values:** `bind`, `call` and `apply` work on builtin functions, and `Buffer.byteLength`, `Buffer.compare`, `Buffer.alloc`, `JSON.parse`, `fs.realpathSync` and `String.prototype` methods can be read as values. `events` and `EventEmitter` can be used as untyped constructor values. +- **JavaScript sources:** + - Bitwise operators convert boolean operands as `ToNumber` does. + - `arguments.length` can be read in sloppy-mode functions with named parameters. + - A function value in an untyped slot may have any signature. + - Prototype methods take their JSDoc parameter types. + - A top-level function declaration that its module reassigns works. + - A `var` can be overwritten with a different kind of value. + - A default-export object literal can hold class values. +- **Index-signature records answer `"k" in record`** for literal keys. A fresh array literal spreads into an array of a wider element type. A `const` annotated with a type that resolves to `any` holds an untyped object. +- **`Set.prototype.add` returns the Set** when its result is used, so `s.add(a).add(b)` compiles. +- **`exec` works on regular expressions held as `any`,** and a numeric enum from a package can be read by a runtime key (`DiagLogLevel[level]`). +- **`dns.lookup` answers that are decided at run time** can be read as untyped values, `net.connect` accepts Node's `lookup` option, and `dns.lookup` passes the `ADDRCONFIG`, `V4MAPPED` and `ALL` hints. +- **`util.inspect`, `console.log` and `util.format` print program-declared `Error` subclasses** with Node's constructor-name header (`[Plain [Error]: m]`). +- **Typed arrays:** `forEach`, `map`, `filter`, `some`, `every`, `find`, `findIndex`, `findLast`, `findLastIndex`, `reduce` and `reduceRight` compile on every typed-array kind. `map` and `filter` return the receiver's kind (a Buffer for a Buffer), and callbacks receive the element, index and array. `BYTES_PER_ELEMENT` reads answer the element size, including on untyped values. Reads and `in` tests of members Node 24's typed-array and Buffer prototypes lack answer `undefined` and `false`, as in Node. Destructuring assignment into typed-array elements compiles. +- **`Math.max` and `Math.min` with spreads among other arguments** (`Math.max(...xs, 5)`) compile, with Node's NaN and ±0 rules. +- **Generic arrow and function-expression bindings inside functions** compile when they use nothing from the enclosing function's frame. +- **`const xs = []` inside a function** takes the element type its later writes give it, as TypeScript evolves it, instead of `number[]`. Bindings whose reads disagree or that are reassigned become dynamic arrays, as at module scope. + +### Performance + +- **Loop counters stay integers in more loops:** counters compared as `i + c` with a bound, and for-loop counters that are reset or copied inside the loop. A record-boundary scan takes 104 ms (Node: 260 ms), and a typed-array scan takes 318 ms (Node: 801 ms). +- **Indexed reads of non-ASCII strings are inline in worker executables,** and strings of up to 2^24 UTF-16 units keep their unit table. A lexer over a 2.7 MB non-ASCII source takes 190 ms (Node: 327 ms). +- **Worker messages:** a thread that finds its mailbox empty spins for up to 20 µs before it blocks. 100,000 round trips between the main thread and a worker take 360 ms, a quarter of Node's time. +- **Local-time `Date` getters cache the zone offset,** UTC getters reuse the last decomposition, and integers 0 to 99 convert to strings without allocating. Local-time date formatting takes 0.81 times Node's time. +- **Untyped code:** `%` and the bitwise operators run on integers for integral operands, `o.x += d` updates a number property in place, and array `length` reads answer from the element count. +- **Sorts by several keys** (comparators that chain field differences with `||`, including optional number fields with a default) sort by key in the runtime instead of calling the comparator, about 24% faster on a benchmark of such sorts. +- **Worker executables start their threads faster:** the allocator marks its memory for core dumps only when the system would write a core, which removes about 680 calls that hold the address-space lock as threads start. tsc-ts checks tRPC 6% faster at 16 checkers. +- **The native compiler walks IR without a closure per node,** about 3 to 10% faster in the frontend and LLVM emission phases of large builds. +- **Full cycle collections that the death rule requests run at the event loop's idle moments.** Watch-mode rebuilds of large projects are 6 to 11% faster. + +### Fixes + +- **`a?.b.c` on an `any` value short-circuits only the guarded step,** as in Node: it is `undefined` when `a` is nullish and throws when `a.b` is undefined. +- **A failed checked cast to an optional record names the deepest failing path,** for example `expected string at $.order.owner.id, got number`. +- **Spreading a non-iterable value held as `any`** throws Node's `x is not iterable` message. +- **`process.versions` entries that the runtime doesn't have read as `undefined`** in projects that use `@types/node`. +- **Functions stored in JavaScript object literals get Node's names,** taken from the property key. +- **`monitorEventLoopDelay().percentile()` returns 511** on an empty histogram, as Node does. +- **A CommonJS `var` statement that mixes `require()` declarators with other bindings** no longer crashes. +- **Untyped async methods called through overrides or untyped receivers return their promises,** which fixes internal compiler errors in programs that use zod's `parseAsync` this way. +- **TypeScript enum exports in compiled packages stay `undefined` until they are assigned,** as in Node. +- **Element writes accept a possibly undefined index** (typed-array writes, compound and update writes, element assignments used as values and destructuring targets) and convert it with ToNumber, as element reads already did. + + + ## 0.4.0 ### Breaking changes @@ -71,8 +144,6 @@ All notable changes to scriptc will be documented in this file. - **Diagnostic code frames window long source lines.** - **npm crashes, internal errors and false TDZ errors** found while compiling popular packages are fixed. - - ## 0.3.6 ### Features diff --git a/packages/cli-darwin-arm64/package.json b/packages/cli-darwin-arm64/package.json index 3c6e73e91..482e5f1e5 100644 --- a/packages/cli-darwin-arm64/package.json +++ b/packages/cli-darwin-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-darwin-arm64", - "version": "0.4.0", + "version": "0.4.1", "description": "Native scriptc compiler for darwin-arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-darwin-x64/package.json b/packages/cli-darwin-x64/package.json index 51121824b..b1ced2c03 100644 --- a/packages/cli-darwin-x64/package.json +++ b/packages/cli-darwin-x64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-darwin-x64", - "version": "0.4.0", + "version": "0.4.1", "description": "Native scriptc compiler for darwin-x64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-linux-arm64-gnu/package.json b/packages/cli-linux-arm64-gnu/package.json index 1273cdfb8..0aa213279 100644 --- a/packages/cli-linux-arm64-gnu/package.json +++ b/packages/cli-linux-arm64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-linux-arm64-gnu", - "version": "0.4.0", + "version": "0.4.1", "description": "Native scriptc compiler for linux-arm64-gnu", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-linux-arm64-musl/package.json b/packages/cli-linux-arm64-musl/package.json index bdba7d919..4ac5434ae 100644 --- a/packages/cli-linux-arm64-musl/package.json +++ b/packages/cli-linux-arm64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-linux-arm64-musl", - "version": "0.4.0", + "version": "0.4.1", "description": "Native scriptc compiler for linux-arm64-musl", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-linux-x64-gnu/package.json b/packages/cli-linux-x64-gnu/package.json index 3de4dcb72..7473c36bc 100644 --- a/packages/cli-linux-x64-gnu/package.json +++ b/packages/cli-linux-x64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-linux-x64-gnu", - "version": "0.4.0", + "version": "0.4.1", "description": "Native scriptc compiler for linux-x64-gnu", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-linux-x64-musl/package.json b/packages/cli-linux-x64-musl/package.json index 97a1fae89..08e048cd1 100644 --- a/packages/cli-linux-x64-musl/package.json +++ b/packages/cli-linux-x64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-linux-x64-musl", - "version": "0.4.0", + "version": "0.4.1", "description": "Native scriptc compiler for linux-x64-musl", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-win32-x64-msvc/package.json b/packages/cli-win32-x64-msvc/package.json index 9dcd31fa2..54e323839 100644 --- a/packages/cli-win32-x64-msvc/package.json +++ b/packages/cli-win32-x64-msvc/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-win32-x64-msvc", - "version": "0.4.0", + "version": "0.4.1", "description": "Native scriptc compiler for win32-x64-msvc", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli/package.json b/packages/cli/package.json index cfe908775..d66ba3247 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,6 +1,6 @@ { "name": "scriptc", - "version": "0.4.0", + "version": "0.4.1", "description": "Compile ordinary TypeScript and JavaScript to small, fast native executables and WebAssembly modules that run without Node.js", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/compiler/package.json b/packages/compiler/package.json index 70bf3b27a..98d984d4f 100644 --- a/packages/compiler/package.json +++ b/packages/compiler/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/compiler", - "version": "0.4.0", + "version": "0.4.1", "description": "The scriptc compiler — TypeScript/JavaScript frontend, typed IR, LLVM backend", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/compiler/surface-manifest.json b/packages/compiler/surface-manifest.json index f5eef72ab..c05e994a8 100644 --- a/packages/compiler/surface-manifest.json +++ b/packages/compiler/surface-manifest.json @@ -1,6 +1,6 @@ { "schemaVersion": 2, - "compilerVersion": "0.4.0", + "compilerVersion": "0.4.1", "coverage": [ "Entries are projected mechanically from the compiler's own decision tables: the diagnostics registry, the unsupported-syntax dispatch tables, the stdlib and node-builtin lowering tables, and the supported-builtin-module list. Nothing is hand-maintained; the manifest regenerates byte-identically from the source tree at this version.", "Absence from this manifest means 'not projected', never 'unsupported'. Surfaces lowered through dedicated code paths rather than tables are not yet projected: console, JSON, Promise/async and the timer surface, the net/http/tls/https/dgram/dns/assert/test/stream/readline module member surfaces, template literals, the regex slice, global functions (parseInt, parseFloat, isNaN, isFinite), and the process surface outside its ambient slice.", diff --git a/packages/experimental/package.json b/packages/experimental/package.json index f32a1085b..5b6906de6 100644 --- a/packages/experimental/package.json +++ b/packages/experimental/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/experimental", - "version": "0.4.0", + "version": "0.4.1", "description": "Experimental scriptc APIs, with Node.js implementations: lifetime regions for ProgramLifetime objects, intern tables shared between threads, and per-thread slots on shared objects", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-darwin-arm64/package.json b/packages/llvm-darwin-arm64/package.json index 5330a5397..472056c39 100644 --- a/packages/llvm-darwin-arm64/package.json +++ b/packages/llvm-darwin-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-darwin-arm64", - "version": "0.4.0", + "version": "0.4.1", "description": "Pinned LLVM code-generation helper for scriptc on macOS arm64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-darwin-x64/package.json b/packages/llvm-darwin-x64/package.json index 6282710b9..7f55cb6b5 100644 --- a/packages/llvm-darwin-x64/package.json +++ b/packages/llvm-darwin-x64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-darwin-x64", - "version": "0.4.0", + "version": "0.4.1", "description": "Pinned LLVM code-generation helper for scriptc on macOS x64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-linux-arm64-gnu/package.json b/packages/llvm-linux-arm64-gnu/package.json index 59b4af11f..0e1a55b22 100644 --- a/packages/llvm-linux-arm64-gnu/package.json +++ b/packages/llvm-linux-arm64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-linux-arm64-gnu", - "version": "0.4.0", + "version": "0.4.1", "description": "Pinned LLVM code-generation helper for scriptc on Linux arm64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-linux-arm64-musl/package.json b/packages/llvm-linux-arm64-musl/package.json index 5ea482253..efdd97444 100644 --- a/packages/llvm-linux-arm64-musl/package.json +++ b/packages/llvm-linux-arm64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-linux-arm64-musl", - "version": "0.4.0", + "version": "0.4.1", "description": "Statically linked LLVM code-generation helper for scriptc on Linux arm64 musl", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-linux-x64-gnu/package.json b/packages/llvm-linux-x64-gnu/package.json index a17b6f393..39667d535 100644 --- a/packages/llvm-linux-x64-gnu/package.json +++ b/packages/llvm-linux-x64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-linux-x64-gnu", - "version": "0.4.0", + "version": "0.4.1", "description": "Pinned LLVM code-generation helper for scriptc on Linux x64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-linux-x64-musl/package.json b/packages/llvm-linux-x64-musl/package.json index 161ca0cb7..9efd41845 100644 --- a/packages/llvm-linux-x64-musl/package.json +++ b/packages/llvm-linux-x64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-linux-x64-musl", - "version": "0.4.0", + "version": "0.4.1", "description": "Statically linked LLVM code-generation helper for scriptc on Linux x64 musl", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-win32-x64-msvc/package.json b/packages/llvm-win32-x64-msvc/package.json index cfcc28a01..329fa308c 100644 --- a/packages/llvm-win32-x64-msvc/package.json +++ b/packages/llvm-win32-x64-msvc/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-win32-x64-msvc", - "version": "0.4.0", + "version": "0.4.1", "description": "Pinned LLVM code-generation helper for scriptc on Windows x64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-android-arm64/package.json b/packages/runtime-android-arm64/package.json index 2d8e22c92..75c54e9c3 100644 --- a/packages/runtime-android-arm64/package.json +++ b/packages/runtime-android-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-android-arm64", - "version": "0.4.0", + "version": "0.4.1", "description": "Precompiled scriptc library runtime pack for Android arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-darwin-arm64/package.json b/packages/runtime-darwin-arm64/package.json index 68c81b246..7473b05f3 100644 --- a/packages/runtime-darwin-arm64/package.json +++ b/packages/runtime-darwin-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-darwin-arm64", - "version": "0.4.0", + "version": "0.4.1", "description": "Precompiled scriptc runtime pack for macOS arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-darwin-x64/package.json b/packages/runtime-darwin-x64/package.json index 54c0d7835..ab8666c88 100644 --- a/packages/runtime-darwin-x64/package.json +++ b/packages/runtime-darwin-x64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-darwin-x64", - "version": "0.4.0", + "version": "0.4.1", "description": "Precompiled scriptc runtime pack for macOS x64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-ios-arm64/package.json b/packages/runtime-ios-arm64/package.json index eb3d0ee27..a3a254685 100644 --- a/packages/runtime-ios-arm64/package.json +++ b/packages/runtime-ios-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-ios-arm64", - "version": "0.4.0", + "version": "0.4.1", "description": "Precompiled scriptc library runtime pack for iOS arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-ios-simulator-arm64/package.json b/packages/runtime-ios-simulator-arm64/package.json index 3b805bb6f..7981422b8 100644 --- a/packages/runtime-ios-simulator-arm64/package.json +++ b/packages/runtime-ios-simulator-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-ios-simulator-arm64", - "version": "0.4.0", + "version": "0.4.1", "description": "Precompiled scriptc library runtime pack for iOS simulator arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-linux-arm64-gnu/package.json b/packages/runtime-linux-arm64-gnu/package.json index ad50f2fe3..d633ac303 100644 --- a/packages/runtime-linux-arm64-gnu/package.json +++ b/packages/runtime-linux-arm64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-linux-arm64-gnu", - "version": "0.4.0", + "version": "0.4.1", "description": "Precompiled scriptc runtime pack for Linux arm64 glibc", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-linux-arm64-musl/package.json b/packages/runtime-linux-arm64-musl/package.json index 4d4c23fdd..460ca25fa 100644 --- a/packages/runtime-linux-arm64-musl/package.json +++ b/packages/runtime-linux-arm64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-linux-arm64-musl", - "version": "0.4.0", + "version": "0.4.1", "description": "Precompiled scriptc runtime pack for Linux arm64 musl", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-linux-x64-gnu/package.json b/packages/runtime-linux-x64-gnu/package.json index ab8780c39..9c6336a04 100644 --- a/packages/runtime-linux-x64-gnu/package.json +++ b/packages/runtime-linux-x64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-linux-x64-gnu", - "version": "0.4.0", + "version": "0.4.1", "description": "Precompiled scriptc runtime pack for Linux x64 glibc", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-linux-x64-musl/package.json b/packages/runtime-linux-x64-musl/package.json index 862b46187..e9f885a5c 100644 --- a/packages/runtime-linux-x64-musl/package.json +++ b/packages/runtime-linux-x64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-linux-x64-musl", - "version": "0.4.0", + "version": "0.4.1", "description": "Precompiled scriptc runtime pack for Linux x64 musl", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-wasm32-wasi/package.json b/packages/runtime-wasm32-wasi/package.json index 48c57d59a..c9a40c2c0 100644 --- a/packages/runtime-wasm32-wasi/package.json +++ b/packages/runtime-wasm32-wasi/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-wasm32-wasi", - "version": "0.4.0", + "version": "0.4.1", "description": "Precompiled scriptc runtime pack for WASI Preview 1", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-win32-x64-msvc/package.json b/packages/runtime-win32-x64-msvc/package.json index fe9ba6892..f7e4c9860 100644 --- a/packages/runtime-win32-x64-msvc/package.json +++ b/packages/runtime-win32-x64-msvc/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-win32-x64-msvc", - "version": "0.4.0", + "version": "0.4.1", "description": "Precompiled scriptc runtime pack for Windows x64 MSVC", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime/package.json b/packages/runtime/package.json index b63f4fad3..1af249b9d 100644 --- a/packages/runtime/package.json +++ b/packages/runtime/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime", - "version": "0.4.0", + "version": "0.4.1", "description": "scriptc native runtime sources and vendored dependencies", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/threads/package.json b/packages/threads/package.json index 358b06dc4..2aeeb7cbb 100644 --- a/packages/threads/package.json +++ b/packages/threads/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/threads", - "version": "0.4.0", + "version": "0.4.1", "description": "Zero-copy publication of immutable object graphs to worker threads (scriptc), with a deep-freezing Node.js implementation", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/types/package.json b/packages/types/package.json index 74d19fc24..ffbbdf4a4 100644 --- a/packages/types/package.json +++ b/packages/types/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/types", - "version": "0.4.0", + "version": "0.4.1", "description": "Contract types for scriptc: type-only annotations (Int32, NoThrow, Immutable) that scriptc checks and optimizes, with no runtime semantics", "license": "Apache-2.0", "homepage": "https://scriptc.dev", From 3f3e90b84618b636094d6c23d80e4c059b96318a Mon Sep 17 00:00:00 2001 From: Malte Ubl Date: Wed, 21 Oct 2026 07:53:10 -0700 Subject: [PATCH 27/51] Drop the EventEmitter method-values follow-up Revert 3a27a92cb ("Forward at most four emit arguments through EventEmitter method values") and 0cf62ebec ("Answer EventEmitter methods read as values off compiled emitters"). Under the scope-B plan, EventEmitter methods read as values, `EventEmitter.prototype.on` in JavaScript and constructor-function emitters belong to Node islands, not the native compiler. The typed/untyped listener sharing and the catchable throw in place of the trap stay. Co-Authored-By: Claude Opus 5.5 --- .../limitations/builtin-apis/eventemitter.mdx | 5 +- .../compiler/src/backend/llvm/lib-shared.ts | 1 - .../lowering/class-dynamic-dispatch.ts | 67 +--- .../lowering/emitter-method-values.ts | 303 ------------------ .../lowering/expressions/property-access.ts | 10 - .../compiler/src/ir/builtin-signatures.ts | 4 - packages/compiler/src/ir/runtime-features.ts | 2 - ...6600-emitter-method-values-untyped.ts.json | 6 - .../6601-emitter-method-values.cjs.json | 6 - .../6602-emitter-method-values-typed.ts.json | 6 - packages/runtime/src/scr_dyn.h | 5 - packages/runtime/src/scr_dyn_builtin_class.c | 45 +-- .../6600-emitter-method-values-untyped.ts | 75 ----- tests/corpus/6601-emitter-method-values.cjs | 73 ----- .../6602-emitter-method-values-typed.ts | 34 -- 15 files changed, 4 insertions(+), 638 deletions(-) delete mode 100644 packages/compiler/src/frontend/lowering/emitter-method-values.ts delete mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6600-emitter-method-values-untyped.ts.json delete mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6601-emitter-method-values.cjs.json delete mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6602-emitter-method-values-typed.ts.json delete mode 100644 tests/corpus/6600-emitter-method-values-untyped.ts delete mode 100644 tests/corpus/6601-emitter-method-values.cjs delete mode 100644 tests/corpus/6602-emitter-method-values-typed.ts diff --git a/docs/content/limitations/builtin-apis/eventemitter.mdx b/docs/content/limitations/builtin-apis/eventemitter.mdx index 8539b814f..0e085c56f 100644 --- a/docs/content/limitations/builtin-apis/eventemitter.mdx +++ b/docs/content/limitations/builtin-apis/eventemitter.mdx @@ -5,9 +5,6 @@ An untyped `emit` of an event that has a listener with typed parameters converts its arguments to the event's typed tuple before any listener runs, and throws a `TypeError` when they do not convert. Through an untyped value, `listeners()` and `rawListeners()` of an event with a listener that has typed parameters are unsupported, and `emit('error')` of an object that is not an Error with no `error` listener is unsupported. Untyped `newListener` and `removeListener` listeners receive `undefined` in place of a listener registered with typed parameters. - An untyped `emit` on, or an untyped read of `emit` from, an instance of a class that overrides `emit` is unsupported. - Calling `EventEmitter.prototype.emit` as a function value on a compiled emitter with more than 4 arguments after the event name is unsupported. - Calling EventEmitter.prototype's methods with an instance of a compiled class that does not extend EventEmitter as the receiver throws a `TypeError`. - Through an untyped value, a stream's `emit` and listener-introspection methods read as `undefined`. + Reading EventEmitter methods as properties through an untyped value (`typeof emitter.listeners`) answers `undefined`. `events.once` accepts EventEmitter and stream instances with a literal event name and an optional object-literal `{ signal }`; a stream's `data` event, other options values, and EventTarget emitters are unsupported. Reading `defaultMaxListeners` through a CommonJS destructured or member binding (`const { defaultMaxListeners } = require("events")`) is unsupported. diff --git a/packages/compiler/src/backend/llvm/lib-shared.ts b/packages/compiler/src/backend/llvm/lib-shared.ts index 3f071d27c..007c78c9e 100644 --- a/packages/compiler/src/backend/llvm/lib-shared.ts +++ b/packages/compiler/src/backend/llvm/lib-shared.ts @@ -1465,7 +1465,6 @@ export const LIB_FN_SYMS: Record = { "dyn.classInherit": "scr_dyn_class_inherit", "dyn.builtinClass": "scr_dyn_builtin_class", "dyn.builtinStreamClass": "scr_dyn_builtin_stream_class", - "dyn.emitterMethod": "scr_dyn_emitter_method", "dyn.builtinInstanceOf": "scr_dyn_builtin_instanceof", "dyn.functionInstanceOf": "scr_dyn_function_instanceof", "dyn.classSuper": "scr_dyn_class_super", diff --git a/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts b/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts index af2a61213..c9e6768a6 100644 --- a/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts +++ b/packages/compiler/src/frontend/lowering/class-dynamic-dispatch.ts @@ -74,12 +74,6 @@ import { lowerDynamicEmitterOn, } from "./lower-event-emitter.js"; import { protocolArm, protocolFieldArm } from "./class-protocol-dispatch.js"; -import { - EMITTER_PROTOTYPE_METHODS, - emitterMethodValue, - inheritedEmitterMethod, - overriddenEmit, -} from "./emitter-method-values.js"; import { STREAM_OPERATORS } from "./lower-stream.js"; import { deletedPropertyNames, fixedFieldFamily } from "./js-class-fields.js"; @@ -2777,7 +2771,6 @@ export class ClassDynamicDispatch { !isClassOwnEnumerableFieldName(dispatch.name) || (!(info.builtinStream !== undefined && streamProperty(dispatch.name)) && !eventTargetInheritedMember(info, dispatch.name) && - !inheritedEmitterMethod(lowerer, info, dispatch.name) && !findMethodOn(lowerer, info, dispatch.name) && !findGenericMethodOn(lowerer, info, dispatch.name) && dispatch.name !== "constructor" && @@ -2896,12 +2889,6 @@ export class ClassDynamicDispatch { loc, }, ]; - // An inherited EventEmitter.prototype method: `in` sees it, the - // own-property probes see only an own property that shadows it. - if (dispatch.probe === "in" && inheritedEmitterMethod(lowerer, info, name)) - return [ - { kind: "return", value: { kind: "boolLit", value: true, type: BOOL, loc }, loc }, - ]; if (!dispatch.probe) return dispatch.reflect ? this.reflectPropertyBody(lowerer, { ...dispatch, name }, info, receiver) @@ -3114,8 +3101,6 @@ export class ClassDynamicDispatch { for (const name of [...STREAM_BOOL_PROPERTIES, ...STREAM_NUM_PROPERTIES]) names.add(name); const inherited = eventTargetApiMembers(info); if (inherited) for (const name of inherited) names.add(name); - for (const name of EMITTER_PROTOTYPE_METHODS.keys()) - if (inheritedEmitterMethod(lowerer, info, name)) names.add(name); for (let owner: ClassInfo | null = info; owner; owner = owner.base) { const methodNames = [...owner.methods.keys()]; @@ -3563,10 +3548,6 @@ export class ClassDynamicDispatch { for (const name of c.methods.keys()) names.add(name.startsWith("get:") || name.startsWith("set:") ? name.slice(4) : name); } - // EventEmitter.prototype's methods on compiled emitters. - for (const { info } of this.propertyReceivers(lowerer, true)) - for (const name of EMITTER_PROTOTYPE_METHODS.keys()) - if (inheritedEmitterMethod(lowerer, info, name)) names.add(name); return names; } @@ -4327,9 +4308,7 @@ export class ClassDynamicDispatch { (!canonical || !seen.has(info.def.name)) && (info === capsule || lowerer.classCanBeConstructed(info)) ) { - // A subclass visited under a runtime capsule (boxed as that - // base) keeps its own canonical plan for values boxed as itself. - if (canonical) seen.add(info.def.name); + seen.add(info.def.name); plans.push({ info, capsule: canonical ? info : capsule, @@ -4749,45 +4728,6 @@ export class ClassDynamicDispatch { loc, }, ]; - // An emit override has no function value (it is specialized per event - // name and payload type). - if ( - !write && - name === "emit" && - overriddenEmit(info) && - inheritedEmitterMethod(lowerer, info, name) - ) - return [ - { - kind: "runtimeFence", - code: "SC2020", - message: - "reading an overridden EventEmitter emit through an untyped value is not supported yet", - loc, - }, - ]; - // An inherited EventEmitter.prototype method: an own property shadows - // it, otherwise the read answers the function - // `EventEmitter.prototype.` reads. - if (!write && inheritedEmitterMethod(lowerer, info, name)) { - const instance = lowerer.coerceToExpected(receiver, DYN); - const bag = this.readPropertyBagOrEmpty(lowerer, instance, loc); - const key: IrExpr = { kind: "strLit", value: name, type: STRING, loc }; - return [ - { - kind: "return", - value: { - kind: "ternary", - cond: this.readBagHas(lowerer, "dyn.hasKey", bag, key, loc), - then: { kind: "libCall", fn: "dyn.bagGet", args: [bag, key, instance], type: DYN, loc }, - else_: emitterMethodValue(lowerer, name, loc), - type: DYN, - loc, - }, - loc, - }, - ]; - } const field = info.fields.get(name); const member = `${write ? "set" : "get"}:${name}`; const accessor = field ? null : findMethodOn(lowerer, info, member); @@ -5219,11 +5159,6 @@ export class ClassDynamicDispatch { loc, }, ]; - // An emit override is specialized per event name and payload type; an - // untyped emit names neither, so it refuses rather than bypassing the - // override for the native emit. - if (method === "emit" && overriddenEmit(info) && inheritedEmitterMethod(lowerer, info, method)) - return fence(); if ( emitterRooted(lowerer, info) && !streamRooted(info) && diff --git a/packages/compiler/src/frontend/lowering/emitter-method-values.ts b/packages/compiler/src/frontend/lowering/emitter-method-values.ts deleted file mode 100644 index 7b4992bee..000000000 --- a/packages/compiler/src/frontend/lowering/emitter-method-values.ts +++ /dev/null @@ -1,303 +0,0 @@ -/** - * EventEmitter.prototype's methods as values. A read of `o.on` on a - * compiled emitter (a program class extending EventEmitter, or - * `new EventEmitter()`) through an untyped value or as a value, and - * `EventEmitter.prototype.on` itself, answer the runtime's - * EventEmitter.prototype functions (scr_dyn_builtin_class.c), the same - * functions JavaScript constructor functions inherit: `on` and - * `addListener` are one function, as are `off` and `removeListener`, and - * each carries Node's `name` and `length`. Called on a compiled emitter - * (`f.call(o, …)`, `f.bind(o)`, `f.apply(o, …)`), such a function hands - * the call to the program's dispatch below, which invokes the method on - * the receiver through the same untyped dispatch a direct `o.on(…)` call - * reaches. - */ -import { BOOL, DYN, F64, STRING, type IrExpr, type IrStmt, type SrcLoc } from "../../ir/ir.js"; -import { dynUndefinedExpr, strLit, varRef } from "../../ir/build.js"; -import * as ts from "../ts7/adapter.js"; -import { locOf } from "../program.js"; -import type { Lowerer } from "./lowerer.js"; -import { findGenericMethodOn, findMethodOn, type ClassInfo } from "./lower-classes.js"; - -/** EventEmitter.prototype's method names. */ -export const EMITTER_PROTOTYPE_METHODS: ReadonlySet = new Set([ - "setMaxListeners", - "getMaxListeners", - "emit", - "addListener", - "on", - "prependListener", - "once", - "prependOnceListener", - "removeListener", - "off", - "removeAllListeners", - "listeners", - "rawListeners", - "listenerCount", - "eventNames", -]); - -/** `emit` forwards this many arguments after the event name; a call with - * more fences. */ -const EMIT_ARGUMENTS = 4; - -/** The runtime's SCR_EE_M_* order: each method's name and the argument - * counts its call forwards. Node's methods that look at - * `arguments.length` (removeAllListeners, emit, listenerCount's optional - * listener) get exactly the arguments given; the rest get their declared - * parameters, an absent one as undefined. */ -const DISPATCHED: readonly { - method: string; - fixed?: number; - counts?: (given: number) => number; -}[] = [ - { method: "addListener", fixed: 2 }, - { method: "prependListener", fixed: 2 }, - { method: "once", fixed: 2 }, - { method: "prependOnceListener", fixed: 2 }, - { method: "removeListener", fixed: 2 }, - { method: "removeAllListeners", counts: (given) => Math.min(given, 1) }, - { method: "emit", counts: (given) => Math.max(given, 1) }, - { method: "listeners", fixed: 1 }, - { method: "rawListeners", fixed: 1 }, - { method: "listenerCount", counts: (given) => (given < 2 ? 1 : 2) }, - { method: "eventNames", fixed: 0 }, - { method: "setMaxListeners", fixed: 1 }, - { method: "getMaxListeners", fixed: 0 }, -]; - -const DISPATCH = "%emitter.method.dispatch"; - -/** An EventEmitter.prototype method a non-stream emitter class inherits - * (no field, method or accessor of that name in its chain; streams keep - * their own surface). An `emit` override is not a method, so `emit` counts - * here too; overriddenEmit separates it. */ -export function inheritedEmitterMethod(lowerer: Lowerer, info: ClassInfo, name: string): boolean { - if (!EMITTER_PROTOTYPE_METHODS.has(name)) return false; - let emitter = false; - for (let c: ClassInfo | null = info; c; c = c.base) { - if (c.builtinStream !== undefined) return false; - if (c.builtinEmitter) emitter = true; - } - return ( - emitter && - !info.fields.has(name) && - !findMethodOn(lowerer, info, name) && - !findGenericMethodOn(lowerer, info, name) && - !findMethodOn(lowerer, info, `get:${name}`) && - !findMethodOn(lowerer, info, `set:${name}`) - ); -} - -/** True when the class or an ancestor overrides `emit` (lowered as - * per-event specializations rather than a method). */ -export function overriddenEmit(info: ClassInfo): boolean { - for (let c: ClassInfo | null = info; c; c = c.base) if (c.emitOverride) return true; - return false; -} - -/** A typed read of an inherited EventEmitter.prototype method as a value - * (`const f = emitter.on`, `typeof emitter.emit`) on a compiled emitter or - * a union of them: the receiver is evaluated, the read answers the - * interned method value. */ -export function lowerEmitterInstanceMethod( - lowerer: Lowerer, - expr: ts.PropertyAccessExpression, -): IrExpr | null { - const name = expr.name.text; - if (!EMITTER_PROTOTYPE_METHODS.has(name) || expr.questionDotToken) return null; - if (ts.isCallExpression(expr.parent) && expr.parent.expression === expr) return null; - const symbol = lowerer.checker.getSymbolAtLocation(expr.name); - if (!symbol || !lowerer.isStdlibSymbol(symbol)) return null; - const receiverType = lowerer.mapTypeOf(lowerer.typeOf(expr.expression)); - if (!receiverType) return null; - const arms = - receiverType.kind === "union" - ? (lowerer.unions.get(receiverType.unionId)?.arms ?? []) - : [receiverType]; - if ( - !arms.length || - !arms.every((arm) => { - const info = arm.kind === "object" ? lowerer.classes.get(arm.className) : undefined; - return info !== undefined && inheritedEmitterMethod(lowerer, info, name); - }) - ) - return null; - const loc = locOf(expr); - return { - kind: "seqExpr", - stmts: [{ kind: "exprStmt", expr: lowerer.lowerExpr(expr.expression), loc }], - result: emitterMethodValue(lowerer, name, loc), - type: DYN, - loc, - }; -} - -/** The program's dispatch for EventEmitter.prototype's functions on - * compiled emitters: `this` is the emitter, the first argument the - * method's index, the rest the call's arguments. */ -function emitterMethodDispatch(lowerer: Lowerer, loc: SrcLoc): IrExpr { - if (!lowerer.liftedFns.some((fn) => fn.name === DISPATCH)) { - const self = varRef("self", DYN, loc); - const values = varRef("values", DYN, loc); - const index = varRef("index", F64, loc); - const given = varRef("given", F64, loc); - const number = (value: number): IrExpr => ({ kind: "numLit", value, type: F64, loc }); - const equals = (left: IrExpr, value: number): IrExpr => ({ - kind: "bin", - op: "===", - left, - right: number(value), - type: BOOL, - loc, - }); - const element = (i: number): IrExpr => ({ - kind: "dynKeyGet", - value: values, - key: lowerer.coerceToExpected(number(i), DYN), - type: DYN, - loc, - }); - const invoke = (method: string, n: number): IrStmt => ({ - kind: "return", - value: { - kind: "dynInvoke", - recv: self, - method, - calleeName: method, - args: Array.from({ length: n }, (_, i) => element(i + 1)), - type: DYN, - loc, - }, - loc, - }); - const body: IrStmt[] = [ - { - kind: "varDecl", - localId: "self", - init: { kind: "libCall", fn: "dyn.this", args: [], type: DYN, loc }, - loc, - }, - { - kind: "varDecl", - localId: "index", - init: { kind: "dynCheck", value: element(0), type: F64, loc }, - loc, - }, - { - kind: "varDecl", - localId: "given", - init: { - kind: "bin", - op: "-", - left: { - kind: "dynCheck", - value: { kind: "dynKeyGet", value: values, key: strLit("length", loc), type: DYN, loc }, - type: F64, - loc, - }, - right: number(1), - type: F64, - loc, - }, - loc, - }, - ]; - DISPATCHED.forEach(({ method, fixed, counts }, i) => { - const arms: IrStmt[] = []; - if (fixed !== undefined || !counts) arms.push(invoke(method, fixed ?? 0)); - else { - // One arm per argument count up to the last one that matters. - const limit = method === "emit" ? EMIT_ARGUMENTS + 1 : 2; - for (let n = 0; n < limit; n++) - arms.push({ - kind: "if", - cond: equals(given, n), - then: [invoke(method, counts(n))], - else_: null, - loc, - }); - if (method === "emit") - arms.push( - { - kind: "if", - cond: equals(given, limit), - then: [invoke(method, limit)], - else_: null, - loc, - }, - { - kind: "runtimeFence", - code: "SC2020", - message: `calling EventEmitter.prototype.emit through a method value with more than ${EMIT_ARGUMENTS} arguments after the event name is not supported yet`, - loc, - }, - ); - else arms.push(invoke(method, counts(limit))); - } - body.push({ kind: "if", cond: equals(index, i), then: arms, else_: null, loc }); - }); - body.push({ kind: "return", value: dynUndefinedExpr(loc), loc }); - lowerer.liftedFns.push({ - name: DISPATCH, - params: [{ localId: "values", name: "values", type: DYN }], - returnType: DYN, - locals: [ - { id: "values", name: "values", type: DYN, mutable: false }, - { id: "self", name: "self", type: DYN, mutable: false }, - { id: "index", name: "index", type: F64, mutable: false }, - { id: "given", name: "given", type: F64, mutable: false }, - ], - body, - loc, - }); - } - return { - kind: "dynFrom", - value: { - kind: "closure", - fnName: DISPATCH, - captures: [], - type: { kind: "func", params: [], ret: DYN, rest: true, argumentsAll: true }, - loc, - }, - type: DYN, - loc, - }; -} - -/** EventEmitter.prototype[name] (DYN). */ -export function emitterMethodValue(lowerer: Lowerer, name: string, loc: SrcLoc): IrExpr { - return { - kind: "libCall", - fn: "dyn.emitterMethod", - args: [{ kind: "strLit", value: name, type: STRING, loc }, emitterMethodDispatch(lowerer, loc)], - type: DYN, - loc, - }; -} - -/** `EventEmitter.prototype.` (any import spelling of the class): - * the interned method value. */ -export function lowerEmitterPrototypeMethod( - lowerer: Lowerer, - expr: ts.PropertyAccessExpression, -): IrExpr | null { - if (!EMITTER_PROTOTYPE_METHODS.has(expr.name.text)) return null; - const prototype = expr.expression; - if ( - !ts.isPropertyAccessExpression(prototype) || - prototype.questionDotToken || - prototype.name.text !== "prototype" - ) - return null; - const owner = prototype.expression; - const symbol = ts.isIdentifier(owner) - ? lowerer.resolveValueSymbol(owner) - : ts.isPropertyAccessExpression(owner) - ? lowerer.checker.getSymbolAtLocation(owner.name) - : null; - if (!lowerer.builtinEmitterInfoOf(symbol)) return null; - return emitterMethodValue(lowerer, expr.name.text, locOf(expr)); -} diff --git a/packages/compiler/src/frontend/lowering/expressions/property-access.ts b/packages/compiler/src/frontend/lowering/expressions/property-access.ts index 1e2722114..36ea0e8c2 100644 --- a/packages/compiler/src/frontend/lowering/expressions/property-access.ts +++ b/packages/compiler/src/frontend/lowering/expressions/property-access.ts @@ -128,10 +128,6 @@ import { lowerReadlineProperty } from "../builtins/readline.js"; import { lowerEventStaticProperty } from "../builtins/numbers.js"; import { lowerStreamProperty, lowerStreamStateProperty, streamSidesOf } from "../lower-stream.js"; import { lowerEventObjectProperty } from "../lower-event-target.js"; -import { - lowerEmitterInstanceMethod, - lowerEmitterPrototypeMethod, -} from "../emitter-method-values.js"; import { iteratorResultValueRead } from "../lower-generators.js"; import { lowerGlobalValue } from "../lower-global-value.js"; import { nullishGenericBindingUnitOf } from "../binding-analysis.js"; @@ -333,12 +329,6 @@ export function lowerPropertyAccessExpr( const ctor = runtimeConstructorReceiver(lowerer, expr); if (ctor) return lowerRuntimeConstructor(lowerer, expr, ctor.info); } - // EventEmitter.prototype's methods as values (emitter-method-values.ts). - { - const method = - lowerEmitterPrototypeMethod(lowerer, expr) ?? lowerEmitterInstanceMethod(lowerer, expr); - if (method) return method; - } // `C.name` through a runtime-provided class VALUE (a `typeof Readable` // parameter): the class object's stored name. if ( diff --git a/packages/compiler/src/ir/builtin-signatures.ts b/packages/compiler/src/ir/builtin-signatures.ts index 9bba57167..d181927fc 100644 --- a/packages/compiler/src/ir/builtin-signatures.ts +++ b/packages/compiler/src/ir/builtin-signatures.ts @@ -3611,10 +3611,6 @@ export const LIB_FN_SIGS = defineLibFnSignatures({ * functions over them get native stream state * (scr_dyn_builtin_stream.c). */ "dyn.builtinStreamClass": { argTypes: [F64], result: DYN }, - /** EventEmitter.prototype[name] off EventEmitter (0) above; the first - * read installs the program's dispatch (the DYN function) that runs - * those methods on compiled emitters. */ - "dyn.emitterMethod": { argTypes: [STRING, DYN], result: DYN }, /** `value instanceof C` for those constructors, over the value's * prototype chain. */ "dyn.builtinInstanceOf": { argTypes: [DYN, F64], result: BOOL }, diff --git a/packages/compiler/src/ir/runtime-features.ts b/packages/compiler/src/ir/runtime-features.ts index 5c38b14f9..fd7c98cf6 100644 --- a/packages/compiler/src/ir/runtime-features.ts +++ b/packages/compiler/src/ir/runtime-features.ts @@ -259,8 +259,6 @@ const BUILTIN_CLASS_LIB_FNS: readonly IrLibFn[] = [ "dyn.builtinClass", // the stream classes, through scr_dyn_builtin_stream.c "dyn.builtinStreamClass", - // EventEmitter.prototype's methods read off compiled emitters - "dyn.emitterMethod", "dyn.builtinInstanceOf", "dyn.functionInstanceOf", ]; diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6600-emitter-method-values-untyped.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6600-emitter-method-values-untyped.ts.json deleted file mode 100644 index 260282e75..000000000 --- a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6600-emitter-method-values-untyped.ts.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "order": [ - "/tests/corpus/6600-emitter-method-values-untyped.ts" - ], - "diags": [] -} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6601-emitter-method-values.cjs.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6601-emitter-method-values.cjs.json deleted file mode 100644 index 13f1aaa7d..000000000 --- a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6601-emitter-method-values.cjs.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "order": [ - "/tests/corpus/6601-emitter-method-values.cjs" - ], - "diags": [] -} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6602-emitter-method-values-typed.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6602-emitter-method-values-typed.ts.json deleted file mode 100644 index 409f1eaa9..000000000 --- a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6602-emitter-method-values-typed.ts.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "order": [ - "/tests/corpus/6602-emitter-method-values-typed.ts" - ], - "diags": [] -} diff --git a/packages/runtime/src/scr_dyn.h b/packages/runtime/src/scr_dyn.h index 14734ab31..401bf2b39 100644 --- a/packages/runtime/src/scr_dyn.h +++ b/packages/runtime/src/scr_dyn.h @@ -1159,11 +1159,6 @@ ScrDyn *scr_dyn_new_class(ScrClassObj *cls, const char *type_key); ScrDyn *scr_dyn_builtin_class(double kind); bool scr_dyn_builtin_instanceof(ScrDyn *value, double kind); bool scr_dyn_function_instanceof(ScrDyn *value, ScrDyn *callee); -/* EventEmitter.prototype[name] (+1); the first call installs DISPATCH, - * the program's function that runs those methods on compiled emitters - * (typed refs): called with the emitter as `this`, the method's - * SCR_EE_M_* index, then the call's arguments. */ -ScrDyn *scr_dyn_emitter_method(const ScrStr *name, ScrDyn *dispatch); /* The stream half of those constructors (scr_dyn_builtin_stream.c, linked * with the dynBuiltinStream feature), registered with * scr_dyn_builtin_class_streams before the first stream class value is diff --git a/packages/runtime/src/scr_dyn_builtin_class.c b/packages/runtime/src/scr_dyn_builtin_class.c index 70a67c559..2f947048a 100644 --- a/packages/runtime/src/scr_dyn_builtin_class.c +++ b/packages/runtime/src/scr_dyn_builtin_class.c @@ -66,35 +66,9 @@ static ScrDyn *bc_class(int kind); /* The stream half, once registered (scr_dyn_builtin_stream.c). */ static SCR_TL const ScrDynStreamOps *bc_stream_ops; -/* The program's dispatch for compiled emitters (a program class extending - * EventEmitter, or `new EventEmitter()`), which reach these methods as - * typed refs whose listeners live in the native registry; installed by - * scr_dyn_emitter_method. */ -static SCR_TL ScrDyn *bc_compiled_dispatch; - -/* method(...args) on a compiled emitter: the dispatch runs with the - * receiver as `this` and the method's SCR_EE_M_* index before the - * arguments. */ -static ScrDyn *bc_compiled_event(int method, ScrDyn *self, ScrDyn *const *args, size_t argc) { - ScrDyn **argv = malloc((argc + 1) * sizeof *argv); - if (!argv) scr_trap("out of memory"); - argv[0] = scr_dyn_new_num(method); - for (size_t i = 0; i < argc; i++) argv[i + 1] = args[i]; - scr_dyn_this_push_dyn(self); - ScrDyn *result = scr_dyn_call(bc_compiled_dispatch, argv, argc + 1, bc_names[BC_EMITTER]); - scr_dyn_this_pop(); - scr_dyn_release(argv[0]); - free(argv); - return result; -} - -/* EventEmitter.prototype's method on a receiver with stream state, or on - * a compiled emitter. */ +/* EventEmitter.prototype's method on a receiver with stream state. */ static bool bc_stream_event(int method, ScrDyn *self, ScrDyn *const *args, size_t argc, ScrDyn **result) { - if (bc_stream_ops != NULL && bc_stream_ops->event(method, self, args, argc, result)) return true; - if (bc_compiled_dispatch == NULL || self->kind != SCR_DYN_TYPED_REF) return false; - *result = bc_compiled_event(method, self, args, argc); - return true; + return bc_stream_ops != NULL && bc_stream_ops->event(method, self, args, argc, result); } /* ── property helpers ───────────────────────────────────────────────── */ @@ -1462,8 +1436,6 @@ static void bc_teardown(void) { if (bc_symbols[i]) scr_dyn_release(bc_symbols[i]); bc_symbols[i] = NULL; } - if (bc_compiled_dispatch) scr_dyn_release(bc_compiled_dispatch); - bc_compiled_dispatch = NULL; if (scr_exc_pending()) scr_exc_clear(); } @@ -1570,19 +1542,6 @@ ScrDyn *scr_dyn_builtin_class(double kind) { return scr_dyn_retain(bc_class(index)); } -/* EventEmitter.prototype[name], +1, read off a compiled emitter or the - * class; the first read installs the program's dispatch for compiled - * emitters (borrowed, retained). */ -ScrDyn *scr_dyn_emitter_method(const ScrStr *name, ScrDyn *dispatch) { - ScrDyn *ctor = bc_class(BC_EMITTER); - if (!bc_compiled_dispatch) bc_compiled_dispatch = scr_dyn_retain(dispatch); - ScrDyn *prototype = bc_get(ctor, "prototype"); - if (!prototype) return NULL; - ScrDyn *method = bc_get(prototype, name->data); - scr_dyn_release(prototype); - return method; -} - /* OrdinaryHasInstance over the value's prototype chain. */ static bool bc_chain_has(ScrDyn *value, ScrDyn *prototype) { if (value->kind != SCR_DYN_OBJ && value->kind != SCR_DYN_ARR && value->kind != SCR_DYN_FUNC && diff --git a/tests/corpus/6600-emitter-method-values-untyped.ts b/tests/corpus/6600-emitter-method-values-untyped.ts deleted file mode 100644 index f25be85d9..000000000 --- a/tests/corpus/6600-emitter-method-values-untyped.ts +++ /dev/null @@ -1,75 +0,0 @@ -// EventEmitter.prototype's methods read as values through untyped -// references to compiled emitters: a program class extending EventEmitter -// and `new EventEmitter()` seen through any and unknown. -import { EventEmitter } from "node:events"; - -class Bus extends EventEmitter { - label = "bus"; - ping(): string { - return `${this.label} ping`; - } -} - -const bus = new Bus(); -const o: any = bus; -console.log(typeof o.on, typeof o.emit, typeof o.listeners, typeof o.ping); -console.log(o.on === o.addListener, o.off === o.removeListener, o.on === o.once); -console.log(o.on === EventEmitter.prototype.on, o.emit === EventEmitter.prototype.emit); -console.log(o.on.name, o.off.name, o.emit.name, o.once.name); -console.log(o.on.length, o.emit.length, o.listeners.length, o.eventNames.length); - -// call, apply and bind with the emitter as the receiver -const add = o.on; -console.log(add.call(o, "a", (x: unknown, y: unknown) => console.log("a:", x, y)) === o); -console.log(o.emit.apply(o, ["a", 1, 2])); -const emit = o.emit.bind(o); -console.log(emit("a", "x"), emit("missing")); -console.log(o.listenerCount.call(o, "a"), o.eventNames.call(o)); -const once = o.once; -once.call(o, "b", () => console.log("b once")); -console.log(emit("b"), emit("b")); -const prepend = o.prependListener; -prepend.call(o, "a", () => console.log("a first")); -emit("a", 3, 4); -console.log(Reflect.apply(o.listenerCount, o, ["a"])); - -// the argument count matters to removeAllListeners and emit -o.on("c", () => {}); -o.on("d", () => {}); -console.log(o.removeAllListeners.call(o, "c") === o, o.eventNames()); -o.removeAllListeners.call(o); -console.log(o.eventNames()); -o.on("many", (...args: unknown[]) => console.log("many:", args.length, args.join(","))); -console.log(emit("many", 1, 2, 3, 4)); -console.log(o.setMaxListeners.call(o, 3) === o, o.getMaxListeners.call(o)); - -// off through a detached method value -const listener = () => console.log("never"); -o.on("e", listener); -const off = o.off; -off.call(o, "e", listener); -console.log(o.emit("e"), o.listenerCount("e")); - -// membership: inherited, not own -console.log("on" in o, "emit" in o, Object.prototype.hasOwnProperty.call(o, "on")); -const key = ["emit", "listeners"][Number(o.listenerCount("a") > 5)]!; -console.log(typeof o[key], o[key] === EventEmitter.prototype.emit); - -// the class's own members still answer -console.log(o.ping(), o.ping.call(o)); - -// new EventEmitter() through unknown -const plain = new EventEmitter(); -const seen: unknown = plain; -const p = seen as any; -const pOn = p.on; -pOn.call(p, "z", (v: unknown) => console.log("z:", v)); -console.log(p.emit.call(p, "z", 9), p.listeners.call(p, "z").length); -console.log(p.on === o.on, p.rawListeners === EventEmitter.prototype.rawListeners); - -// a detached method called without a receiver throws a TypeError -try { - add("x", () => {}); -} catch (err) { - console.log("detached:", err instanceof TypeError); -} diff --git a/tests/corpus/6601-emitter-method-values.cjs b/tests/corpus/6601-emitter-method-values.cjs deleted file mode 100644 index e462bda8e..000000000 --- a/tests/corpus/6601-emitter-method-values.cjs +++ /dev/null @@ -1,73 +0,0 @@ -"use strict"; -// EventEmitter.prototype's methods as values in JavaScript: read off a -// compiled subclass, off `new EventEmitter()`, off EventEmitter.prototype -// by name, and shared with constructor-function emitters. -const EventEmitter = require("events"); -const util = require("util"); - -class Queue extends EventEmitter { - constructor() { - super(); - this.items = []; - } - push(v) { - this.items.push(v); - this.emit("push", v); - return this; - } -} - -const q = new Queue(); -const on = q.on; -on.call(q, "push", (v) => console.log("pushed", v)); -q.push(1).push(2); -for (const m of ["on", "addListener", "emit", "off", "once", "prependOnceListener", "listeners", "listenerCount", "eventNames"]) { - console.log(m, typeof q[m], q[m] === EventEmitter.prototype[m], q[m].name, q[m].length); -} - -// EventEmitter.prototype's functions called on compiled emitters -const proto = EventEmitter.prototype; -proto.on.call(q, "drain", () => console.log("drained")); -console.log(proto.emit.call(q, "drain"), proto.listenerCount.call(q, "push")); -console.log(proto.eventNames.call(q)); -proto.removeAllListeners.call(q, "drain"); -console.log(q.eventNames(), Reflect.apply(proto.listenerCount, q, ["drain"])); - -// new EventEmitter() -const plain = new EventEmitter(); -const emit = plain.emit; -plain.on("hi", (a, b, c) => console.log("hi", a, b, c)); -console.log(emit.call(plain, "hi", 1, "two", [3])); -console.log(emit.call(plain, "hi", 1, 2, 3, 4)); -const fn = () => console.log("never"); -plain.once("x", fn); -const off = plain.off; -off.call(plain, "x", fn); -console.log(plain.emit("x"), plain.listenerCount("x")); -const bound = plain.prependListener.bind(plain); -bound("hi", () => console.log("hi first")); -plain.emit("hi", "a"); - -// a constructor-function emitter shares the same functions -function Legacy() { - EventEmitter.call(this); -} -util.inherits(Legacy, EventEmitter); -const legacy = new Legacy(); -console.log(legacy.on === q.on, legacy.emit === plain.emit); -legacy.on("tick", (n) => console.log("legacy tick", n)); -q.on.call(legacy, "tick", (n) => console.log("legacy tick again", n)); -console.log(plain.emit.call(legacy, "tick", 7), legacy.listenerCount("tick")); - -// method values of a class expression whose instances escape -function makeEmitter() { - return new (class extends EventEmitter { - fire() { - return this.emit("fire", "local"); - } - })(); -} -const local = makeEmitter(); -const localOn = local.on; -localOn.call(local, "fire", (v) => console.log("fire", v)); -console.log(local.fire(), "on" in local, Object.keys(local).includes("on")); diff --git a/tests/corpus/6602-emitter-method-values-typed.ts b/tests/corpus/6602-emitter-method-values-typed.ts deleted file mode 100644 index 45fb7f1fc..000000000 --- a/tests/corpus/6602-emitter-method-values-typed.ts +++ /dev/null @@ -1,34 +0,0 @@ -// EventEmitter.prototype's methods read as values from typed references -// and from EventEmitter.prototype in TypeScript. -import { EventEmitter } from "node:events"; -import * as events from "node:events"; - -class Counter extends EventEmitter { - count = 0; - bump(): void { - this.count++; - this.emit("bump", this.count); - } -} - -const counter = new Counter(); -const plain = new EventEmitter(); -console.log(typeof counter.on, typeof plain.emit, typeof EventEmitter.prototype.listeners); -console.log(counter.on === plain.on, counter.addListener === EventEmitter.prototype.on); -console.log(events.EventEmitter.prototype.off === plain.removeListener); - -const onValue: unknown = counter.on; -const emitValue: unknown = plain.emit; -console.log(typeof onValue, onValue === EventEmitter.prototype.addListener, emitValue === counter.emit); - -const register = counter.on as any; -register.call(counter, "bump", (n: number) => console.log("bump", n)); -counter.bump(); -EventEmitter.prototype.on.call(counter, "bump", (n: number) => console.log("bump again", n)); -counter.bump(); -console.log(EventEmitter.prototype.listenerCount.call(counter, "bump")); -console.log(EventEmitter.prototype.emit.call(plain, "nothing")); - -const pick = (emitter: EventEmitter, name: "on" | "emit"): unknown => - name === "on" ? emitter.on : emitter.emit; -console.log(pick(counter, "on") === pick(plain, "on"), pick(counter, "emit") === pick(plain, "on")); From 6fd37b33c5470901f7b2558cf527cbfdc48202af Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 09:55:28 -0500 Subject: [PATCH 28/51] Mark compact header loads invariant only where the header cannot change --- .../compiler/src/backend/llvm/classes.test.ts | 23 ++++++++++++++++++- packages/compiler/src/backend/llvm/classes.ts | 23 ++++++++++++------- 2 files changed, 37 insertions(+), 9 deletions(-) diff --git a/packages/compiler/src/backend/llvm/classes.test.ts b/packages/compiler/src/backend/llvm/classes.test.ts index 2f554b458..5daaf4300 100644 --- a/packages/compiler/src/backend/llvm/classes.test.ts +++ b/packages/compiler/src/backend/llvm/classes.test.ts @@ -1,6 +1,12 @@ import { expect, test } from "vitest"; import { DYN, F64, STRING, VOID, type IrModule } from "../../ir/ir.js"; -import { buildClassGraph, classMembershipIntervals, emitClassMembershipHelper } from "./classes.js"; +import { + buildClassGraph, + classIdLoadLines, + classMembershipIntervals, + emitClassMembershipHelper, + type LlClassMeta, +} from "./classes.js"; import { emitLlvmModule } from "./emitter.js"; import { mangleClassNew, @@ -214,6 +220,21 @@ test("a recycled block is cleared from the end of a ProgramLifetime instance's 3 } }); +test("invariant class id reads of compact instances leave out the published bit", () => { + const group = ", !invariant.group !{}"; + const compact = (post: number) => ({ compact: true, root: { post } }) as unknown as LlClassMeta; + // Publication sets the header word's top bit; ids below 2^16 are read + // from the low 16 bits, which never change after construction. + expect(classIdLoadLines(compact(40), "%o", "%id", group)).toEqual([ + " %id.h = load i16, ptr %o, !invariant.group !{}", + " %id = zext i16 %id.h to i32", + ]); + // Wider ids, and reads without the metadata, mask the whole word. + const masked = [" %id.w = load i32, ptr %o", " %id = and i32 %id.w, 268435455"]; + expect(classIdLoadLines(compact(0xffff), "%o", "%id", group)).toEqual(masked); + expect(classIdLoadLines(compact(40), "%o", "%id", "")).toEqual(masked); +}); + test("published ProgramLifetime instances use the published mark every other object uses", () => { const node = { kind: "object" as const, className: "Node" }; const leaf = { kind: "object" as const, className: "Leaf" }; diff --git a/packages/compiler/src/backend/llvm/classes.ts b/packages/compiler/src/backend/llvm/classes.ts index 9963fad5b..03ee8caa1 100644 --- a/packages/compiler/src/backend/llvm/classes.ts +++ b/packages/compiler/src/backend/llvm/classes.ts @@ -526,18 +526,24 @@ export function classIdVtLines( /** Lines that load the class id of the LlClassMeta.idInRc instance `obj` * into the i32 `id`: the high half of the rc word, or the low 28 bits of a - * compact instance's header word (compactHeaderWord). */ + * compact instance's header word (compactHeaderWord). `group` is the + * metadata of the load (`, !invariant.group !{}` for a load LLVM may treat + * as invariant), and it only goes on bytes that never change: publication + * sets the top bit of a compact header word, so a compact instance's id is + * then read as the word's low 16 bits, which hold the whole id when the + * hierarchy's ids are below 2^16, and otherwise as the masked word without + * the metadata. */ export function classIdLoadLines( meta: LlClassMeta, obj: string, id: string, group: string, ): string[] { - if (meta.compact) - return [ - ` ${id}.w = load i32, ptr ${obj}${group}`, - ` ${id} = and i32 ${id}.w, ${COMPACT_ID_MASK}`, - ]; + if (meta.compact) { + if (group !== "" && meta.root.post + 1 <= 0xffff) + return [` ${id}.h = load i16, ptr ${obj}${group}`, ` ${id} = zext i16 ${id}.h to i32`]; + return [` ${id}.w = load i32, ptr ${obj}`, ` ${id} = and i32 ${id}.w, ${COMPACT_ID_MASK}`]; + } return [ ` ${id}p = getelementptr inbounds i8, ptr ${obj}, i64 ${CLASS_ID_OFFSET}`, ` ${id} = load i32, ptr ${id}p${group}`, @@ -1115,8 +1121,9 @@ export function emitClassShapes( nw.push(` store i32 ${compactHeaderWord(meta)}, ptr %o ; no count (compactHeaderWord)`); else if (meta.compact) { // The header word: the class id and no count (compactHeaderWord). - // Only publication writes it, setting COMPACT_PUBLISHED, which the - // class id reads mask off, so they keep !invariant.group. + // Only publication writes it again, setting COMPACT_PUBLISHED in the + // top byte, so the class id reads that carry !invariant.group read + // the low 16 bits only (classIdLoadLines). host.declare(`declare ptr @llvm.launder.invariant.group.p0(ptr)`); nw.push( ` %o.vt = call ptr @llvm.launder.invariant.group.p0(ptr %o)`, From 523d5ea7884c6027c0b6f461998b94dfb9afff41 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 09:56:30 -0500 Subject: [PATCH 29/51] Compute emitter module facts once per module The emitter's analyses repeated whole-module work. dynamicFieldStores walked every function five times on the emitted module (the emitter constructor, the int32 slot analysis, the final field analysis and both cycle mutability analyses), findInitializerBindings ran twice, functionCallFacts walked every function three times, the int32 slot analysis walked every function four more times before its fixpoint, and the runtime feature scan ran two to four times. The constructor now computes dynamic field stores and initializer bindings once and passes them to the analyses, which compute them themselves when a caller does not. functionCallFacts and the int32 slot analysis collect their call, arity, box and reader facts in one walk per function and apply the facts that depend on earlier steps, in walk order, where those steps complete. The emitter reads one runtime feature scan, taken the first time any feature is needed. Emitted LLVM is unchanged. --- .../compiler/src/backend/cycle-analysis.ts | 11 +- .../compiler/src/backend/cycle-mutability.ts | 5 +- packages/compiler/src/backend/immutability.ts | 8 +- .../compiler/src/backend/llvm/emit-module.ts | 3 +- packages/compiler/src/backend/llvm/emitter.ts | 34 +-- packages/compiler/src/ir/final-fields.ts | 10 +- packages/compiler/src/ir/int32-slots.ts | 212 ++++++++++-------- 7 files changed, 156 insertions(+), 127 deletions(-) diff --git a/packages/compiler/src/backend/cycle-analysis.ts b/packages/compiler/src/backend/cycle-analysis.ts index c41b88067..aa624579f 100644 --- a/packages/compiler/src/backend/cycle-analysis.ts +++ b/packages/compiler/src/backend/cycle-analysis.ts @@ -1,3 +1,4 @@ +import type { DynamicFieldStores } from "../ir/int32-slots.js"; import type { IrModule, IrType } from "../ir/ir.js"; import { DYN, @@ -34,8 +35,11 @@ import { computeCycleMutability, fieldKey } from "./cycle-mutability.js"; * subclass and retain touches the cycle header, so header presence must be * uniform across an extends tree): a unit is cycle-capable iff ANY member * is — every backend uses the same grouping. */ -export function computeTraced(mod: IrModule): { shapes: Set; unions: Set } { - const { shapes, unions } = analyzeCycles(mod, false); +export function computeTraced( + mod: IrModule, + stores?: DynamicFieldStores, +): { shapes: Set; unions: Set } { + const { shapes, unions } = analyzeCycles(mod, false, stores); return { shapes, unions }; } @@ -88,6 +92,7 @@ export function explainCycle(mod: IrModule, key: string): CycleExplanation | nul function analyzeCycles( mod: IrModule, labels: boolean, + stores?: DynamicFieldStores, ): { shapes: Set; unions: Set; @@ -120,7 +125,7 @@ function analyzeCycles( const lifetimeRoots = new Set(mod.contracts?.programLifetime ?? []); const programLifetime = (className: string): boolean => lifetimeRoots.size > 0 && lifetimeRoots.has(rootOf(className)); - const mutability = computeCycleMutability(mod, unitOfClass); + const mutability = computeCycleMutability(mod, unitOfClass, stores); const shapeDefs = [ ...classes.map((c) => ({ key: `object:${c.name}`, diff --git a/packages/compiler/src/backend/cycle-mutability.ts b/packages/compiler/src/backend/cycle-mutability.ts index 79993fdff..78daa6204 100644 --- a/packages/compiler/src/backend/cycle-mutability.ts +++ b/packages/compiler/src/backend/cycle-mutability.ts @@ -1,6 +1,6 @@ import type { IrExpr, IrFunction, IrModule, IrStmt, IrType } from "../ir/ir.js"; import { RUNTIME_EMITTER_CLASS, RUNTIME_STREAM_CLASSES, typeKey } from "../ir/ir.js"; -import { dynamicFieldStores } from "../ir/int32-slots.js"; +import { dynamicFieldStores, type DynamicFieldStores } from "../ir/int32-slots.js"; import { everyExprChild, everyStmtChild, everyTypeChild } from "../ir/traverse.js"; /** Which shape fields can be written after their object escapes. @@ -72,6 +72,7 @@ export function fieldKey(unit: string, field: string): string { export function computeCycleMutability( mod: IrModule, unitOfClass: (className: string) => string, + stores?: DynamicFieldStores, ): CycleMutability { const result: CycleMutability = { all: false, @@ -100,7 +101,7 @@ export function computeCycleMutability( // intrinsics other than publish (which only marks objects immortal), the // intern-table operations (which store published values) and ThreadSlots // reads stay conservative. - const dynStores = dynamicFieldStores(mod); + const dynStores = stores ?? dynamicFieldStores(mod); const dynamicStores = dynStores.all || dynStores.names.size > 0; result.emptyPropertyBags = !dynamicStores; const boundaryWrites = (e: IrExpr): boolean => diff --git a/packages/compiler/src/backend/immutability.ts b/packages/compiler/src/backend/immutability.ts index 949aec75e..6c37731e3 100644 --- a/packages/compiler/src/backend/immutability.ts +++ b/packages/compiler/src/backend/immutability.ts @@ -1,5 +1,6 @@ /* Static immutability facts the backend uses to drop run-time immutability * checks (see computeConstructionFacts). Pure functions of the IR module. */ +import type { DynamicFieldStores } from "../ir/int32-slots.js"; import type { IrExpr, IrModule, IrStmt } from "../ir/ir.js"; import { everyExprChild, everyStmtChild } from "../ir/traverse.js"; import { @@ -44,7 +45,10 @@ export class ImmutableFields { } } -export function computeImmutableFields(mod: IrModule): ImmutableFields | null { +export function computeImmutableFields( + mod: IrModule, + stores?: DynamicFieldStores, +): ImmutableFields | null { if (mod.lib !== undefined) return null; const baseOf = new Map((mod.classes ?? []).map((c) => [c.name, c.base ?? null] as const)); const roots = new Map(); @@ -58,7 +62,7 @@ export function computeImmutableFields(mod: IrModule): ImmutableFields | null { return cur; }; const unitOfClass = (name: string): string => `object:${rootOf(name)}`; - const mutability = computeCycleMutability(mod, unitOfClass); + const mutability = computeCycleMutability(mod, unitOfClass, stores); return mutability.all ? null : new ImmutableFields(mutability, unitOfClass); } diff --git a/packages/compiler/src/backend/llvm/emit-module.ts b/packages/compiler/src/backend/llvm/emit-module.ts index e5c8c5d6d..732a21ee6 100644 --- a/packages/compiler/src/backend/llvm/emit-module.ts +++ b/packages/compiler/src/backend/llvm/emit-module.ts @@ -29,7 +29,6 @@ import { vAdapters, } from "./shapes.js"; import { emitLlvmLayouts } from "./layouts.js"; -import { moduleRuntimeFeatures } from "../../ir/runtime-features.js"; import { emitArraySearchHelpers } from "./array-search.js"; import { emitClassMembershipHelper } from "./classes.js"; import { type RegexMatcherCode, compileRegexMatcher } from "./regex-matchers.js"; @@ -138,7 +137,7 @@ export function emitModuleParts(emitter: LlEmitter): string[] { // outlive the RC audit — release it with the globals (the runtime ABI’s // sc_release_globals tail). Only when the dispatch unit is even // linked (defineProps is the only writer). - const runtimeFeatures = moduleRuntimeFeatures(emitter.mod); + const runtimeFeatures = emitter.runtimeFeatures; const fnValueProps = [...emitter.fnValues]; if (fnValueProps.length > 0) emitter.declare(`declare void @scr_box_release(ptr)`); if (emitter.classObjs.size > 0) diff --git a/packages/compiler/src/backend/llvm/emitter.ts b/packages/compiler/src/backend/llvm/emitter.ts index 18cee0501..514bf61b4 100644 --- a/packages/compiler/src/backend/llvm/emitter.ts +++ b/packages/compiler/src/backend/llvm/emitter.ts @@ -54,11 +54,7 @@ import { VOID, classJsName, } from "../../ir/ir.js"; -import { - moduleRuntimeFeatures, - moduleUsesDynBuiltinStream, - moduleUsesMockTimers, -} from "../../ir/runtime-features.js"; +import { moduleRuntimeFeatures, type RuntimeFeatures } from "../../ir/runtime-features.js"; import { captureFreeFunctions, observesStacks } from "../stack-observations.js"; import { type DenseArrayLoop } from "./dense-array-loops.js"; import { findInitializedByteLoopBindings } from "./byte-windows.js"; @@ -879,13 +875,16 @@ export class LlEmitter { /** The program calls mock.timers members (moduleUsesMockTimers): timer * calls emit scr_test.c's mock-aware entry points. */ readonly mockTimers: boolean; - private schedulesInContextValue: boolean | undefined = undefined; + private runtimeFeaturesValue: RuntimeFeatures | undefined; + /** The module's runtime features (moduleRuntimeFeatures), found with one + * walk of the module the first time any of them is read. */ + get runtimeFeatures(): RuntimeFeatures { + return (this.runtimeFeaturesValue ??= moduleRuntimeFeatures(this.mod)); + } /** Whether timer and tick scheduling goes through the *_als entry * points (RuntimeFeatures asyncContext). */ get schedulesInContext(): boolean { - if (this.schedulesInContextValue === undefined) - this.schedulesInContextValue = moduleRuntimeFeatures(this.mod).asyncContext; - return this.schedulesInContextValue; + return this.runtimeFeatures.asyncContext; } /** Emitted ref-kind resolve thunks for new Promise, interned per inner * typeKey → thunk symbol. */ @@ -901,10 +900,9 @@ export class LlEmitter { /** Identity-preserving static→dyn capsules for program classes and Web * APIs whose values remain directly observable. */ readonly liveDynRefAdapters = new Map(); - private dynBuiltinStreamUse: boolean | undefined; /** See moduleUsesDynBuiltinStream. */ get usesDynBuiltinStream(): boolean { - return (this.dynBuiltinStreamUse ??= moduleUsesDynBuiltinStream(this.mod)); + return this.runtimeFeatures.dynBuiltinStream; } /** Runtime-arm dispatchers for live union values. */ readonly liveDynUnionRefAdapters = new Map(); @@ -1141,7 +1139,7 @@ export class LlEmitter { ) { this.stackTraces = observesStacks(mod); this.nullableUnions = new NullableUnions(mod, options.debugSources === undefined); - this.mockTimers = moduleUsesMockTimers(mod); + this.mockTimers = this.runtimeFeatures.mockTimers; this.debug = options.debugSources === undefined ? null @@ -1154,14 +1152,18 @@ export class LlEmitter { ); this.constantNumericTables = findConstantNumericTables(mod); this.initializerBindings = findInitializerBindings(mod); - this.int32Slots = analyzeInt32Slots(mod); + // One walk finds the dynamic field stores every analysis below consults. const dynamicStores = dynamicFieldStores(mod); + this.int32Slots = analyzeInt32Slots(mod, { + dynamicStores, + initializerBindings: this.initializerBindings, + }); this.liveViewsReadOnly = mod.lib === undefined && !dynamicStores.all && dynamicStores.names.size === 0; this.int32Abis = this.debug === null ? findInt32Abis(mod, this.int32Slots) : new Map(); this.immortalGlobals = new Set(mod.contracts?.immortalGlobals ?? []); this.programLifetime = new Set(mod.contracts?.programLifetime ?? []); - this.finalFields = analyzeFinalFields(mod); + this.finalFields = analyzeFinalFields(mod, false, dynamicStores); this.sizeType = options.pointerBits === 32 ? "i32" : "i64"; this.ffiExtendNarrowIntegers = options.wasi === true || ffiExtendsNarrowIntegers(options.targetTriple); @@ -1231,7 +1233,7 @@ export class LlEmitter { this.unframedFunctions = this.stackTraces ? captureFreeFunctions(mod, this.stackChecks, this.publishedTypes) : new Set(); - this.immutableFields = computeImmutableFields(mod); + this.immutableFields = computeImmutableFields(mod, dynamicStores); this.constructorFunctions = new Set((mod.classes ?? []).map((c) => `%${c.name}.constructor`)); this.indirectMayThrow = mt.indirect || mod.workers === true; for (const cls of mod.classes ?? []) { @@ -1356,7 +1358,7 @@ export class LlEmitter { this.receiverReaders = new AmbientReceiverReaders(this.fnByName); this.stackCaptures = new StackCaptures(this.fnByName, this.callLifetimes, this.stackCallbacks); this.lazyCaptures = new LazyCaptures(this.fnByName); - const traced = computeTraced(mod); + const traced = computeTraced(mod, dynamicStores); this.tracedShapes = traced.shapes; this.tracedUnions = traced.unions; // A shape union's value is whichever record it holds, and the collector diff --git a/packages/compiler/src/ir/final-fields.ts b/packages/compiler/src/ir/final-fields.ts index 21572c910..b0340cbaa 100644 --- a/packages/compiler/src/ir/final-fields.ts +++ b/packages/compiler/src/ir/final-fields.ts @@ -1,5 +1,5 @@ import { isRefCounted, type IrClassDef, type IrExpr, type IrModule, type IrStmt } from "./ir.js"; -import { dynamicFieldStores } from "./int32-slots.js"; +import { dynamicFieldStores, type DynamicFieldStores } from "./int32-slots.js"; import { everyExprChild, everyStmtChild } from "./traverse.js"; /** Whole-program final reference fields. @@ -78,11 +78,15 @@ export function constructorClass(fnName: string): string | null { /** `scalars` extends the analysis from reference-counted fields to fields of * every type, for consumers that read a final field's value once instead of * borrowing it (field-values.ts). */ -export function analyzeFinalFields(mod: IrModule, scalars = false): FinalFields { +export function analyzeFinalFields( + mod: IrModule, + scalars = false, + stores?: DynamicFieldStores, +): FinalFields { const classes = new Map((mod.classes ?? []).map((c) => [c.name, c])); const finals = new FinalFields(classes); if (mod.lib) return finals; - const dynamic = dynamicFieldStores(mod); + const dynamic = stores ?? dynamicFieldStores(mod); if (dynamic.all) return finals; const genericFamilies = new Set(); diff --git a/packages/compiler/src/ir/int32-slots.ts b/packages/compiler/src/ir/int32-slots.ts index a08693013..781fa86bd 100644 --- a/packages/compiler/src/ir/int32-slots.ts +++ b/packages/compiler/src/ir/int32-slots.ts @@ -1,4 +1,4 @@ -import type { IrClassDef, IrExpr, IrFunction, IrModule, IrStmt, IrType } from "./ir.js"; +import type { IrClassDef, IrExpr, IrFunction, IrGlobal, IrModule, IrStmt, IrType } from "./ir.js"; import { analyzeIntegerRanges, INT32_RANGE, @@ -458,11 +458,18 @@ function walkFunction( * whose key is not a literal), otherwise the * literal keys of `dyn.keySet`/`dyn.keyDelete`. A view commits a store back * into the instance's native field slots, which the IR does not show. */ +export interface DynamicFieldStores { + all: boolean; + names: Set; +} + +/** Computes DynamicFieldStores with one walk of every function. Analyses of + * the same module accept the result so the emitter walks the module once. */ export function dynamicFieldStores( mod: IrModule, records = new Map((mod.records ?? []).map((r) => [r.id, r])), unions = new Map((mod.unions ?? []).map((u) => [u.id, u])), -): { all: boolean; names: Set } { +): DynamicFieldStores { const dynTypes = new Map(); const dynNamed = new Map(); const dynish = (type: IrType): boolean => { @@ -522,7 +529,11 @@ export function dynamicFieldStores( * can write: the class reaches checked-dynamic code (directly or through * another exposed type), and some dynamic store in the program can name the * field. */ -export function dynamicallyStoredFamilies(mod: IrModule, slots: Int32Slots): Set { +export function dynamicallyStoredFamilies( + mod: IrModule, + slots: Int32Slots, + stores?: DynamicFieldStores, +): Set { const classes = new Map((mod.classes ?? []).map((c) => [c.name, c])); const children = new Map(); for (const cls of classes.values()) @@ -588,7 +599,8 @@ export function dynamicallyStoredFamilies(mod: IrModule, slots: Int32Slots): Set // writes every field from the view, which holds the exact snapshot value // of any property no store named. So reaching dynamic code disqualifies // exactly the field names some dynamic store in the program can name. - const { all: dynWritesAll, names: dynWritten } = dynamicFieldStores(mod, records, unions); + const { all: dynWritesAll, names: dynWritten } = + stores ?? dynamicFieldStores(mod, records, unions); for (const fn of mod.functions) walkFunction(fn, { expr: (e) => { @@ -655,20 +667,11 @@ export function functionCallFacts( const runtimePrefixes: string[] = []; for (const cls of classes.values()) if (runtimeRooted(cls.name)) runtimePrefixes.push(`%${cls.name}.`); - for (const fn of mod.functions) - walkFunction(fn, { - expr: (e) => { - const referenced = exprFunctionReference(e); - if (referenced !== undefined) escaped.add(referenced); - if (e.kind !== "virtualCall") return; - for (const name of subtree(e.className)) virtualTargets.add(`%${name}.${e.method}`); - for (let c = classes.get(e.className); c; c = c.base ? classes.get(c.base) : undefined) - virtualTargets.add(`%${c.name}.${e.method}`); - }, - }); // A class value can reach `new` through its construct thunk; one that - // only feeds an instanceof interval check cannot. + // only feeds an instanceof interval check cannot. Class references are + // checked once every function has contributed its tested-only ones. const testedOnly = new Set(); + const classRefs: (IrExpr & { kind: "classRef" })[] = []; for (const fn of mod.functions) { // Direct operands, and unboxed locals bound once to a class reference // and read only as instanceof operands (the lowering's temporary). @@ -681,11 +684,18 @@ export function functionCallFacts( }; walkFunction(fn, { expr: (e) => { - if (e.kind === "instanceOfValue") { + const referenced = exprFunctionReference(e); + if (referenced !== undefined) escaped.add(referenced); + if (e.kind === "virtualCall") { + for (const name of subtree(e.className)) virtualTargets.add(`%${name}.${e.method}`); + for (let c = classes.get(e.className); c; c = c.base ? classes.get(c.base) : undefined) + virtualTargets.add(`%${c.name}.${e.method}`); + } else if (e.kind === "instanceOfValue") { if (e.classValue.kind === "classRef") testedOnly.add(e.classValue); else if (e.classValue.kind === "varRef") count(tests, e.classValue.localId); } else if (e.kind === "varRef") count(reads, e.localId); else if (e.kind === "assignExpr" || e.kind === "incDec") count(writes, e.localId); + else if (e.kind === "classRef") classRefs.push(e); }, stmt: (s) => { if (s.kind === "varDecl" && s.init?.kind === "classRef") bound.set(s.localId, s.init); @@ -698,18 +708,19 @@ export function functionCallFacts( if (!boxed.has(id) && writes.get(id) === 1 && reads.get(id) === tests.get(id)) testedOnly.add(ref); } - for (const fn of mod.functions) - walkFunction(fn, { - expr: (e) => { - if (e.kind === "classRef" && !testedOnly.has(e)) escaped.add(`%${e.className}.constructor`); - }, - }); + for (const e of classRefs) if (!testedOnly.has(e)) escaped.add(`%${e.className}.constructor`); const runtimeVirtual = (name: string): boolean => runtimePrefixes.some((prefix) => name.startsWith(prefix)); return { valueEscaped: escaped, virtualTargets, runtimeVirtual }; } -export function analyzeInt32Slots(mod: IrModule): Int32Slots { +/** Facts about the same module that a caller has already computed. */ +export interface Int32SlotInputs { + dynamicStores?: DynamicFieldStores; + initializerBindings?: ReadonlyMap; +} + +export function analyzeInt32Slots(mod: IrModule, inputs: Int32SlotInputs = {}): Int32Slots { const classes = new Map((mod.classes ?? []).map((c) => [c.name, c])); const slots = new Int32Slots(classes); if (mod.lib) return slots; @@ -780,7 +791,7 @@ export function analyzeInt32Slots(mod: IrModule): Int32Slots { } /* ── dynamic, reflective and runtime exposure ── */ - for (const family of dynamicallyStoredFamilies(mod, slots)) { + for (const family of dynamicallyStoredFamilies(mod, slots, inputs.dynamicStores)) { if (slots.isPinnedField(family)) continue; dropFamily(family, "dynamic stores can name the field"); } @@ -799,56 +810,16 @@ export function analyzeInt32Slots(mod: IrModule): Int32Slots { const dropAllParams = (fn: IrFunction, why: string): void => { for (const param of fn.params) dropParam(fn.name, param.localId, why); }; - // Only functions some IR call reaches directly get parameter facts: one - // the backend invokes by convention has no call site to prove against. + // One walk of every function collects what the steps below need: the + // functions IR calls reach directly, calls whose arity differs from their + // callee's, the captured bindings' boxes, and which functions read which + // slot facts. Facts that depend on earlier steps are applied in walk order + // where those steps complete. const called = slots.called; - for (const fn of mod.functions) - walkFunction(fn, { - expr: (e) => { - if (e.kind === "call") called.add(e.callee); - else if (e.kind === "new") called.add(`%${e.className}.constructor`); - }, - }); - for (const fn of mod.functions) { - if (fn.returnType.kind === "f64" && !fn.async && !fn.generator) { - slots.addReturn(fn.name); - slots.assumeNonNegative(`ret:${fn.name}`); - } - if ( - fn.async || - fn.generator || - !called.has(fn.name) || - escaped.has(fn.name) || - virtual(fn.name) - ) - continue; - for (const param of fn.params) - if (param.type.kind === "f64" && !slots.isPinned(paramKey(fn.name, param.localId))) { - slots.addParam(fn.name, param.localId); - slots.assumeNonNegative(paramKey(fn.name, param.localId)); - } - } + const arityMismatches: IrFunction[] = []; const arityChecked = (fn: IrFunction | undefined, args: number, offset: number): void => { - if (fn && fn.params.length !== args + offset) dropAllParams(fn, "call arity differs"); + if (fn && fn.params.length !== args + offset) arityMismatches.push(fn); }; - for (const fn of mod.functions) - walkFunction(fn, { - expr: (e) => { - if (e.kind === "call") arityChecked(functions.get(e.callee), e.args.length, 0); - else if (e.kind === "new") - arityChecked(functions.get(`%${e.className}.constructor`), e.args.length, 1); - }, - }); - - // Candidates the fixpoint may drop; rangedParameters retries some. - const candidates = new Map(); - const retry = new Set(); - for (const fn of mod.functions) { - const ids = fn.params.filter((p) => slots.hasParam(fn.name, p.localId)).map((p) => p.localId); - if (ids.length > 0) candidates.set(fn.name, ids); - } - - /* ── captured bindings ── */ // A captured local lives in one box shared by its declaring function and // every closure that captures it (closure.captures[i] is the lifted // function's captures[i]). The box holds an int32 when every write to it @@ -864,6 +835,16 @@ export function analyzeInt32Slots(mod: IrModule): Int32Slots { const boxKey = (fnName: string, localId: string): string => `${fnName}\0${localId}`; const boxMembers = new Set(); const unstableBoxes = new Set(); + // Readers: which functions depend on which slot facts. Reads of a boxed + // local resolve to its box once the boxes are known. + const readers = new Map>(); + const reads = (slot: string, fnName: string): void => { + const set = readers.get(slot); + if (set) set.add(fnName); + else readers.set(slot, new Set([fnName])); + }; + const boxedReads: [IrFunction, Set][] = []; + const incrementedFamilies: string[] = []; for (const fn of mod.functions) { for (const local of fn.locals) if (local.boxed) { @@ -873,9 +854,27 @@ export function analyzeInt32Slots(mod: IrModule): Int32Slots { } for (const capture of fn.classCaptures ?? []) unstableBoxes.add(boxKey(fn.name, capture.localId)); + const read = new Set(); walkFunction(fn, { expr: (e) => { - if (e.kind === "closure") { + if (e.kind === "call") { + called.add(e.callee); + arityChecked(functions.get(e.callee), e.args.length, 0); + reads(`ret:${e.callee}`, fn.name); + } else if (e.kind === "new") { + const constructor = `%${e.className}.constructor`; + called.add(constructor); + arityChecked(functions.get(constructor), e.args.length, 1); + // Constructors return nothing: this only records their callers. + reads(`ret:${constructor}`, fn.name); + } else if (e.kind === "varRef") read.add(e.localId); + else if (e.kind === "fieldGet") { + const family = slots.family(e.className, e.field); + if (family) reads(`field:${family}`, fn.name); + } else if (e.kind === "fieldIncDec") { + const family = slots.family(e.className, e.field); + if (family) incrementedFamilies.push(family); + } else if (e.kind === "closure") { const lifted = functions.get(e.fnName); e.captures.forEach((id, i) => { const inner = lifted?.captures?.[i]; @@ -897,7 +896,41 @@ export function analyzeInt32Slots(mod: IrModule): Int32Slots { unstableBoxes.add(boxKey(fn.name, s.catchLocalId)); }, }); + if (read.size > 0) boxedReads.push([fn, read]); } + + // Only functions some IR call reaches directly get parameter facts: one + // the backend invokes by convention has no call site to prove against. + for (const fn of mod.functions) { + if (fn.returnType.kind === "f64" && !fn.async && !fn.generator) { + slots.addReturn(fn.name); + slots.assumeNonNegative(`ret:${fn.name}`); + } + if ( + fn.async || + fn.generator || + !called.has(fn.name) || + escaped.has(fn.name) || + virtual(fn.name) + ) + continue; + for (const param of fn.params) + if (param.type.kind === "f64" && !slots.isPinned(paramKey(fn.name, param.localId))) { + slots.addParam(fn.name, param.localId); + slots.assumeNonNegative(paramKey(fn.name, param.localId)); + } + } + for (const fn of arityMismatches) dropAllParams(fn, "call arity differs"); + + // Candidates the fixpoint may drop; rangedParameters retries some. + const candidates = new Map(); + const retry = new Set(); + for (const fn of mod.functions) { + const ids = fn.params.filter((p) => slots.hasParam(fn.name, p.localId)).map((p) => p.localId); + if (ids.length > 0) candidates.set(fn.name, ids); + } + + /* ── captured bindings ── */ const groups = new Map(); for (const key of boxMembers) { const root = boxRoot(key); @@ -960,34 +993,15 @@ export function analyzeInt32Slots(mod: IrModule): Int32Slots { return slots.dropBox(group); }; - /* ── readers: which functions depend on which slot facts ── */ - const readers = new Map>(); - const reads = (slot: string, fnName: string): void => { - const set = readers.get(slot); - if (set) set.add(fnName); - else readers.set(slot, new Set([fnName])); - }; - for (const fn of mod.functions) - walkFunction(fn, { - expr: (e) => { - if (e.kind === "fieldGet") { - const family = slots.family(e.className, e.field); - if (family) reads(`field:${family}`, fn.name); - } else if (e.kind === "call") reads(`ret:${e.callee}`, fn.name); - // Constructors return nothing: this only records their callers. - else if (e.kind === "new") reads(`ret:%${e.className}.constructor`, fn.name); - else if (e.kind === "varRef") { - const group = slots.boxGroup(fn.name, e.localId); - if (group !== null) reads(`box:${group}`, fn.name); - } else if (e.kind === "fieldIncDec") { - const family = slots.family(e.className, e.field); - if (family) dropFamily(family, "++/-- on the field"); - } - }, - }); + for (const [fn, read] of boxedReads) + for (const localId of read) { + const group = slots.boxGroup(fn.name, localId); + if (group !== null) reads(`box:${group}`, fn.name); + } + for (const family of incrementedFamilies) dropFamily(family, "++/-- on the field"); /* ── the optimistic fixpoint ── */ - const bindings = findInitializerBindings(mod); + const bindings = inputs.initializerBindings ?? findInitializerBindings(mod); const numeric = (fn: IrFunction): IrFunction => withInitializerBindings(fn, bindings.get(fn.name) ?? []); const queue: string[] = mod.functions.map((fn) => fn.name); From 94f040b3df83559e8561dd586d07edfc8ae24b40 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 09:57:56 -0500 Subject: [PATCH 30/51] Address review comments on Buffer methods for checked-dynamic values --- .../src/frontend/lowering/containers/bytes.ts | 2 + packages/runtime/src/scr_bytes.c | 64 ++++++- packages/runtime/src/scr_bytes_dyn.c | 169 +++++++++++++++--- tests/corpus/6930-dyn-buffer-methods.cjs | 122 +++++++++++++ .../6930-dyn-buffer-methods.cjs.json | 2 +- 5 files changed, 328 insertions(+), 31 deletions(-) diff --git a/packages/compiler/src/frontend/lowering/containers/bytes.ts b/packages/compiler/src/frontend/lowering/containers/bytes.ts index 8d1d17821..46ddb1dbb 100644 --- a/packages/compiler/src/frontend/lowering/containers/bytes.ts +++ b/packages/compiler/src/frontend/lowering/containers/bytes.ts @@ -1479,6 +1479,8 @@ function lowerBufferInstanceMethod( if (encName === "utf16le") align = 2; } else if (vT?.kind === "bytes" && vT.elem === "u8") { needle = lowerer.lowerExpr(args[0]!); + // A Buffer needle under utf16le also searches whole 2-byte units. + if (encName === "utf16le") align = 2; } else { lowerer.noLowering( `.${name} of '${lowerer.checker.typeToString(lowerer.typeOf(args[0]!))}' values`, diff --git a/packages/runtime/src/scr_bytes.c b/packages/runtime/src/scr_bytes.c index 6262b2530..69e26e5d7 100644 --- a/packages/runtime/src/scr_bytes.c +++ b/packages/runtime/src/scr_bytes.c @@ -1489,8 +1489,10 @@ ScrStr *scr_strdec_end(const ScrStr *enc, double pending) { static void scr_bytes_to_str_bounds(const ScrBytes *b, double start, double end, size_t *s0_out, size_t *e0_out) { size_t len = b->len; - size_t s0 = start <= 0 ? 0 : ((size_t)start > len ? len : (size_t)start); - size_t e0 = end <= 0 ? 0 : ((size_t)end > len ? len : (size_t)end); + /* Clamped before the conversion: NaN and values at or below 0 are 0, + * values at or past the length (Infinity included) are the length. */ + size_t s0 = !(start > 0) ? 0 : start >= (double)len ? len : (size_t)start; + size_t e0 = !(end > 0) ? 0 : end >= (double)len ? len : (size_t)end; if (e0 < s0) e0 = s0; *s0_out = s0; *e0_out = e0; @@ -3204,15 +3206,61 @@ double scr_bytes_compare(const ScrBytes *src, const ScrBytes *target, double nar return sn < tn ? -1 : 1; } +/* node_buffer.cc's IndexOfBuffer/IndexOfString for the UCS2 encoding over + * an encoded needle: buffer.js clamps the byteOffset to the int32 range, + * IndexOfOffset picks the start, the search runs over whole 2-byte units of + * the haystack (an odd trailing byte of either side is ignored), and a match + * is reported at its byte position. */ +static double scr_bytes_index_of_ucs2(const ScrBytes *haystack, const ScrBytes *needle, double at, bool fwd) { + int64_t length = (int64_t)haystack->len, needle_length = (int64_t)needle->len; + /* NaN (an omitted or non-numeric byteOffset) searches everything. */ + if (at > 2147483647.0) at = 2147483647.0; + else if (at < -2147483648.0) at = -2147483648.0; + int64_t offset_i64 = at != at ? (fwd ? 0 : length) : (int64_t)trunc(at); + int64_t opt_offset; + if (offset_i64 < 0) { + if (offset_i64 + length >= 0) opt_offset = length + offset_i64; + else opt_offset = fwd || needle_length == 0 ? 0 : -1; + } else if (offset_i64 + needle_length <= length) { + opt_offset = offset_i64; + } else if (needle_length == 0) { + opt_offset = length; + } else { + opt_offset = fwd ? -1 : length - 1; + } + size_t search_end = haystack->len & ~(size_t)1; + if (needle_length == 0) return (double)(opt_offset < (int64_t)search_end ? opt_offset : (int64_t)search_end); + if (length == 0 || opt_offset <= -1) return -1; + size_t offset = (size_t)opt_offset; + if (offset >= search_end) { + if (fwd || search_end == 0) return -1; + offset = search_end - 1; + } + if ((fwd && needle->len + offset > search_end) || needle->len > search_end) return -1; + if (search_end < 2 || needle->len < 2) return -1; + size_t units = search_end / 2, needle_units = needle->len / 2, start = offset / 2; + if (units < needle_units) return -1; + size_t last = units - needle_units; + if (fwd) { + for (size_t p = start; p <= last; p++) + if (!memcmp(haystack->data + 2 * p, needle->data, 2 * needle_units)) return (double)(2 * p); + return -1; + } + for (size_t p = start < last ? start : last;; p--) { + if (!memcmp(haystack->data + 2 * p, needle->data, 2 * needle_units)) return (double)(2 * p); + if (p == 0) return -1; + } +} + /* The shared search core: needle bytes, a coerced byteOffset (trunc; NaN * means 'search everything' — 0 forward, the end backward; negatives * count from the end, and a still-negative backward search answers -1), - * and the utf16le alignment stride. */ + * and the alignment: 2 searches utf16le units (scr_bytes_index_of_ucs2). */ double scr_bytes_index_of(const ScrBytes *b, const ScrBytes *needle, double off, double align, bool fwd) { SCR_BYTES_SNAPSHOT(b); SCR_BYTES_SNAPSHOT(needle); + if (align == 2) return scr_bytes_index_of_ucs2(b, needle, off, fwd); size_t len = b->len, nlen = needle->len; - size_t step = align == 2 ? 2 : 1; double o = (off != off) ? (fwd ? 0 : (double)len) : trunc(off); if (o < 0) { o += (double)len; @@ -3225,17 +3273,15 @@ double scr_bytes_index_of(const ScrBytes *b, const ScrBytes *needle, double off, if (nlen > len) return -1; if (fwd) { size_t start = o > (double)len ? len : (size_t)o; - if (step == 2) start += start % 2; - for (size_t i = start; i + nlen <= len; i += step) { + for (size_t i = start; i + nlen <= len; i++) { if (memcmp(b->data + i, needle->data, nlen) == 0) return (double)i; } return -1; } size_t start = o > (double)(len - nlen) ? len - nlen : (size_t)o; - if (step == 2) start -= start % 2; - for (size_t i = start;; i -= step) { + for (size_t i = start;; i--) { if (memcmp(b->data + i, needle->data, nlen) == 0) return (double)i; - if (i < step) break; + if (i == 0) break; } return -1; } diff --git a/packages/runtime/src/scr_bytes_dyn.c b/packages/runtime/src/scr_bytes_dyn.c index 58130a072..280aa423c 100644 --- a/packages/runtime/src/scr_bytes_dyn.c +++ b/packages/runtime/src/scr_bytes_dyn.c @@ -187,6 +187,91 @@ static ScrStr *scr_bytes_dyn_quoted(const ScrStr *text) { return scr_jb_finish(&jb); } +/* Appends Node's inspect of an ArrayBuffer at indentation level `indent`: + * the first 100 bytes as hex pairs, on one line when it fits the 80-column + * break length (reduceToSingleString), one entry per line otherwise. */ +static void scr_bytes_dyn_inspect_array_buffer(ScrJsonBuf *jb, const ScrBytes *owner, size_t indent) { + static const char digits[] = "0123456789abcdef"; + size_t len = owner->len * scr_bytes_elem_size(owner->elem); + size_t shown = len < 100 ? len : 100; + ScrJsonBuf contents; + scr_jb_init(&contents); + scr_jb_puts(&contents, "[Uint8Contents]: <"); + for (size_t i = 0; i < shown; i++) { + uint8_t byte; + scr_bytes_read(owner, i, &byte, 1); + if (i) scr_jb_putc(&contents, ' '); + scr_jb_putc(&contents, digits[byte >> 4]); + scr_jb_putc(&contents, digits[byte & 15]); + } + if (len > shown) { + char more[48]; + snprintf(more, sizeof more, " ... %zu more byte%s", len - shown, len - shown > 1 ? "s" : ""); + scr_jb_puts(&contents, more); + } + scr_jb_putc(&contents, '>'); + ScrStr *first = scr_jb_finish(&contents); + char second[40]; + int second_len = snprintf(second, sizeof second, "[byteLength]: %zu", len); + /* isBelowBreakLength with two entries, start = 2 + indentation + "ArrayBuffer {" + 10. */ + size_t total = 2 + (2 + indent + 13 + 10) + first->len + (size_t)second_len; + if (total <= 80) { + scr_jb_puts(jb, "ArrayBuffer { "); + scr_jb_put_str(jb, first); + scr_jb_puts(jb, ", "); + scr_jb_puts(jb, second); + scr_jb_puts(jb, " }"); + } else { + scr_jb_puts(jb, "ArrayBuffer {\n"); + for (size_t i = 0; i < indent + 2; i++) scr_jb_putc(jb, ' '); + scr_jb_put_str(jb, first); + scr_jb_puts(jb, ",\n"); + for (size_t i = 0; i < indent + 2; i++) scr_jb_putc(jb, ' '); + scr_jb_puts(jb, second); + scr_jb_putc(jb, '\n'); + for (size_t i = 0; i < indent; i++) scr_jb_putc(jb, ' '); + scr_jb_putc(jb, '}'); + } + scr_str_release(first); +} + +/* ERR_INVALID_ARG_VALUE for an empty ArrayBufferView fill value: Node's + * inspect of the view (a Buffer, an empty typed array, or a DataView with + * its buffer), cut to 128 characters followed by "...". */ +static ScrStr *scr_bytes_dyn_empty_view_message(const ScrDyn *value) { + const ScrBytes *view = value->v.bytes; + ScrJsonBuf jb; + scr_jb_init(&jb); + if (value->buffer) { + scr_jb_puts(&jb, ""); + } else if (!view->is_data_view) { + scr_jb_puts(&jb, scr_bytes_elem_name(view->elem)); + scr_jb_puts(&jb, "(0) []"); + } else { + const ScrBytes *owner = view->backing ? view->backing : view; + char line[64]; + snprintf(line, sizeof line, "DataView {\n [byteLength]: 0,\n [byteOffset]: %zu,\n [buffer]: ", + (size_t)(view->data - owner->data)); + scr_jb_puts(&jb, line); + scr_bytes_dyn_inspect_array_buffer(&jb, owner, 2); + scr_jb_puts(&jb, "\n}"); + } + ScrStr *inspected = scr_jb_finish(&jb); + ScrJsonBuf out; + scr_jb_init(&out); + scr_jb_puts(&out, "The argument 'value' is invalid. Received "); + if (inspected->len > 128) { + ScrStr *head = scr_str_new(inspected->data, 128); + scr_jb_put_str(&out, head); + scr_str_release(head); + scr_jb_puts(&out, "..."); + } else { + scr_jb_put_str(&out, inspected); + } + scr_str_release(inspected); + return scr_jb_finish(&out); +} + /* buf.fill(value[, offset[, end]][, encoding]): buffer.js's _fill. A string * takes its encoding from the first string among offset and end; numbers and * other non-view values fill with ToUint32(value) & 255. Answers the receiver. */ @@ -244,9 +329,23 @@ static ScrDyn *scr_bytes_dyn_fill(ScrDyn *recv, ScrDyn *const *args, size_t argc /* An empty string fills with zeros. */ out = text->len == 0 ? scr_bytes_fill_num(b, 0, 2, from, to) : scr_bytes_fill(b, pattern, 2, from, to); scr_bytes_release(pattern); - } else if (value->kind == SCR_DYN_BYTES && !value->v.bytes->is_data_view && - value->v.bytes->elem == SCR_BYTES_U8) { - out = scr_bytes_fill(b, value->v.bytes, 2, from, to); + } else if (value->kind == SCR_DYN_BYTES) { + /* Any ArrayBufferView repeats its raw bytes: a byte view over the + * value's storage (a typed array of wider elements, or a DataView). */ + const ScrBytes *view = value->v.bytes; + if (view->len == 0) { + ScrStr *message = scr_bytes_dyn_empty_view_message(value); + scr_throw_error_msg_code(SCR_ERR_TYPE, message->data, message->len, "ERR_INVALID_ARG_VALUE"); + scr_str_release(message); + return NULL; + } + ScrBytes pattern = { .rc = 1, + .len = view->len * scr_bytes_elem_size(view->elem), + .elem = SCR_BYTES_U8, + .data = view->data, + .backing = NULL, + .shared = view->shared }; + out = scr_bytes_fill(b, &pattern, 2, from, to); } else { double n; if (!scr_dyn_number_coerce_js(value, &n)) return NULL; @@ -256,21 +355,27 @@ static ScrDyn *scr_bytes_dyn_fill(ScrDyn *recv, ScrDyn *const *args, size_t argc scr_bytes_release(out); return scr_dyn_retain(recv); } -/* buf.toString(encoding, start, end): the window clamps to - * [0, length] with NaN as 0, an empty window answers "" before the - * encoding resolves, and an undefined encoding is utf8. */ +/* buf.toString(encoding, start, end): the window clamps to [0, length] + * with NaN as 0 (infinite and out-of-range values included), an empty + * window answers "" before the encoding resolves, and an undefined encoding + * is utf8. */ static ScrDyn *scr_bytes_dyn_to_string(ScrBytes *b, ScrDyn *const *args, size_t argc) { const ScrDyn *encoding = scr_bytes_dyn_arg(args, argc, 0); const ScrDyn *start = scr_bytes_dyn_arg(args, argc, 1); const ScrDyn *end = scr_bytes_dyn_arg(args, argc, 2); - double from = 0, to = (double)b->len; + const double len = (double)b->len; + double from = NAN, to = len; if (!scr_bytes_dyn_undef(start) && !scr_dyn_number_coerce_js(start, &from)) return NULL; if (!scr_bytes_dyn_undef(end) && !scr_dyn_number_coerce_js(end, &to)) return NULL; - if (from != from) from = 0; - if (to != to) to = 0; - from = trunc(from); - to = trunc(to); - if (from >= (double)b->len || to <= from || to <= 0) return scr_bytes_dyn_str("", 0); + /* `if (start <= 0) start = 0; else if (start >= len) return ''; else + * start = MathTrunc(start) || 0`, then `end > len` clamps to the length + * and other ends truncate (NaN as 0), so both stay within [0, len]. */ + if (from <= 0) from = 0; + else if (from >= len) return scr_bytes_dyn_str("", 0); + else from = from == from ? trunc(from) : 0; + if (to > len) to = len; + else to = to == to ? trunc(to) : 0; + if (to <= from) return scr_bytes_dyn_str("", 0); ScrStr *enc = scr_bytes_dyn_undef(encoding) ? scr_str_new("utf8", 4) : scr_bytes_dyn_encoding(encoding); if (!enc) return NULL; ScrStr *text = scr_bytes_to_str_range(b, enc, from, to); @@ -313,6 +418,23 @@ static ScrDyn *scr_bytes_dyn_copy(const ScrBytes *b, ScrDyn *const *args, size_t return scr_dyn_new_num(scr_bytes_copy_into(b, target->v.bytes, 3, offsets[0], offsets[1], offsets[2])); } +/* Whether an encoding name is one of the utf16le spellings, in any ASCII case. */ +static bool scr_bytes_dyn_is_ucs2(const ScrStr *raw) { + static const char *const names[] = { "utf16le", "utf-16le", "ucs2", "ucs-2" }; + for (size_t i = 0; i < sizeof names / sizeof names[0]; i++) { + size_t n = strlen(names[i]); + if (raw->len != n) continue; + size_t k = 0; + for (; k < n; k++) { + char c = raw->data[k]; + if (c >= 'A' && c <= 'Z') c = (char)(c + ('a' - 'A')); + if (c != names[i][k]) break; + } + if (k == n) return true; + } + return false; +} + /* buf.indexOf/lastIndexOf/includes(value[, byteOffset][, encoding]): * buffer.js's bidirectionalIndexOf. A string byteOffset is the encoding; * the offset clamps to the int32 range, and NaN searches everything. */ @@ -333,25 +455,30 @@ static ScrDyn *scr_bytes_dyn_index_of(const ScrBytes *b, ScrDyn *const *args, si if (value->kind == SCR_DYN_NUM) { found = scr_bytes_index_of_num(b, value->v.num, at, fwd); } else { - ScrStr *enc = NULL; + ScrStr *raw = NULL; if (!scr_bytes_dyn_undef(encoding)) { - ScrStr *raw = scr_dyn_string_coerce_js(encoding); + raw = scr_dyn_string_coerce_js(encoding); if (!raw) return NULL; - /* An unknown encoding throws for a string needle only. */ - enc = value->kind == SCR_DYN_STR ? scr_bytes_normalize_encoding(raw) : NULL; - scr_str_release(raw); - if (value->kind == SCR_DYN_STR && !enc) return NULL; } if (value->kind == SCR_DYN_STR) { - if (!enc) enc = scr_str_new("utf8", 4); + /* An unknown encoding throws for a string needle. */ + ScrStr *enc = raw ? scr_bytes_normalize_encoding(raw) : scr_str_new("utf8", 4); + if (raw) scr_str_release(raw); + if (!enc) return NULL; + bool ucs2 = enc->len == 7 && !memcmp(enc->data, "utf16le", 7); ScrBytes *needle = scr_bytes_from_str(value->v.str, enc); - found = scr_bytes_index_of(b, needle, at, enc->len == 7 && !memcmp(enc->data, "utf16le", 7) ? 2 : 1, fwd); + found = scr_bytes_index_of(b, needle, at, ucs2 ? 2 : 1, fwd); scr_bytes_release(needle); scr_str_release(enc); } else if (value->kind == SCR_DYN_BYTES && !value->v.bytes->is_data_view && value->v.bytes->elem == SCR_BYTES_U8) { - found = scr_bytes_index_of(b, value->v.bytes, at, 1, fwd); + /* A Buffer needle searches 2-byte units under a utf16le encoding and + * bytes under any other, an unknown name included. */ + bool ucs2 = raw && scr_bytes_dyn_is_ucs2(raw); + if (raw) scr_str_release(raw); + found = scr_bytes_index_of(b, value->v.bytes, at, ucs2 ? 2 : 1, fwd); } else { + if (raw) scr_str_release(raw); scr_dyn_arg_type_fail("value", "one of type number or string or an instance of Buffer or Uint8Array", value); return NULL; } diff --git a/tests/corpus/6930-dyn-buffer-methods.cjs b/tests/corpus/6930-dyn-buffer-methods.cjs index d9870d274..5167a3188 100644 --- a/tests/corpus/6930-dyn-buffer-methods.cjs +++ b/tests/corpus/6930-dyn-buffer-methods.cjs @@ -143,3 +143,125 @@ console.log(box.text.indexOf("an"), box.text.lastIndexOf("an"), box.text.include console.log(box.list.indexOf(2), box.list.includes(4), box.list.fill(0, 1).join(","), box.list.toString()); console.log(Array.from(box.bytes.fill(9, 2)), box.bytes.toString()); console.log(box.sink.write("x"), box.sink.written, box.sink.toString(1, 2)); + +// fill with other ArrayBufferViews repeats their raw bytes; an empty view is +// ERR_INVALID_ARG_VALUE with Node's inspect of the view. +const tm = (name, f) => { + try { + const v = f(); + console.log(name, Buffer.isBuffer(v) ? `<${v.toString("hex")}>` : JSON.stringify(v)); + } catch (e) { + console.log(name, e.name, e.code, JSON.stringify(e.message)); + } +}; +function Held(bytes) { + this.b = bytes; +} +const held = (bytes) => new Held(bytes).b; +const filled = (n) => held(Buffer.alloc(n)); +tm("fill Uint16Array", () => filled(6).fill(new Uint16Array([0x0102]))); +tm("fill DataView", () => filled(6).fill(new DataView(new Uint8Array([7, 8, 9]).buffer))); +tm("fill DataView window", () => filled(6).fill(new DataView(new Uint8Array([1, 2, 3, 4]).buffer, 1, 2))); +tm("fill Float64Array", () => filled(10).fill(new Float64Array([1]))); +tm("fill subarray", () => filled(6).fill(new Uint16Array([1, 2, 3]).subarray(1))); +tm("fill Int8Array range", () => filled(6).fill(new Int8Array([-1, 2]), 1, 4)); +tm("fill empty Uint16Array", () => filled(6).fill(new Uint16Array(0))); +tm("fill empty Uint8Array", () => filled(6).fill(new Uint8Array(0))); +for (const [size, offset] of [[0, 0], [1, 1], [6, 6], [7, 7], [8, 8], [120, 3], [101, 101]]) + tm(`fill empty DataView ${size}/${offset}`, () => filled(6).fill(new DataView(new ArrayBuffer(size), offset, 0))); +tm("fill empty DataView over bytes", () => filled(6).fill(new DataView(new Uint8Array(200).fill(171).buffer, 7, 0))); + +// Buffer needles under utf16le search whole 2-byte units. +const units = held(Buffer.from("abcabc", "utf16le")); +const split = held(Buffer.from([0x61, 0x00, 0x62, 0x00, 0x00, 0x62, 0x00, 0x63])); +const N = (bytes) => Buffer.from(bytes); +tm("indexOf utf16le needle", () => [split.indexOf(N([0x62, 0x00]), 0, "ucs2"), split.indexOf(N([0x62, 0x00]))]); +tm("indexOf utf16le odd position", () => [ + split.indexOf(N([0x00, 0x62]), 0, "ucs2"), + split.indexOf(N([0x00, 0x62]), 0, "UTF-16LE"), + split.indexOf(N([0x00, 0x62])), +]); +tm("lastIndexOf utf16le needle", () => [ + units.lastIndexOf(N([0x62, 0x00]), undefined, "ucs2"), + split.lastIndexOf(N([0x00, 0x62]), undefined, "ucs2"), + ...[5, 3, 2, 1, -100, 100].map((at) => split.lastIndexOf(N([0x62, 0x00]), at, "ucs2")), +]); +tm("indexOf utf16le offsets", () => [1, 2, 3, -5, 7, 2.9].map((at) => split.indexOf(N([0x62, 0x00]), at, "ucs2"))); +tm("indexOf utf16le short needles", () => [ + split.indexOf(N([0x62]), 0, "ucs2"), + held(N([0x62])).indexOf(N([0x62]), 0, "ucs2"), + split.indexOf(Buffer.alloc(0), 3, "ucs2"), + split.indexOf(Buffer.alloc(0), 30, "ucs2"), + split.indexOf(N([0x62, 0, 0]), 0, "ucs2"), +]); +tm("indexOf utf16le odd lengths", () => [ + held(N([1, 0, 2, 0, 3])).indexOf(N([2, 0, 3]), 0, "ucs2"), + held(N([1, 0, 2, 0, 3, 0])).indexOf(N([2, 0, 3]), 0, "ucs2"), + held(N([1, 2, 1, 2, 1, 2])).indexOf(N([2, 1, 2]), 0, "ucs2"), + held(N([1, 0, 2, 0, 3])).lastIndexOf(N([3]), undefined, "ucs2"), + held(N([0, 2, 0, 2, 0])).lastIndexOf(N([2, 0]), undefined, "ucs2"), +]); +tm("indexOf buffer needle other encodings", () => [ + split.indexOf(N([0x62, 0x00]), 0, "nope"), + split.indexOf(N([0x00, 0x62]), 0, "nope"), + split.indexOf(N([0x00, 0x62]), 0, "latin1"), +]); +tm("indexOf utf16le strings", () => [ + split.indexOf("b", 0, "ucs2"), + units.lastIndexOf("b", undefined, "ucs2"), + units.indexOf("b", 3, "ucs2"), + units.lastIndexOf("b", 3, "ucs2"), + units.indexOf("", 3, "ucs2"), + units.indexOf("", 30, "ucs2"), + units.lastIndexOf("", 30, "ucs2"), + split.indexOf("戀", 0, "ucs2"), +]); +tm("includes utf16le needle", () => [split.includes(N([0x00, 0x62]), 0, "ucs2"), split.includes(N([0x62, 0x00]), 3, "ucs2")]); + +// toString windows with infinite and out-of-range bounds. +const word = held(Buffer.from("hello")); +tm("toString infinite bounds", () => [ + word.toString("utf8", 1, Infinity), + word.toString("utf8", -Infinity, 2), + word.toString("utf8", Infinity), + word.toString("utf8", NaN, 3), + word.toString("utf8", 1e300, 2), + word.toString("utf8", 1, 1e300), + word.toString("utf8", -1e300, 1e300), + word.toString("utf8", 0, -Infinity), + word.toString("utf8", 2, NaN), + word.toString("utf8", 0.5, 4.5), + word.toString("hex", -0.5, -0), + word.toString("latin1", 4.9), + word.toString(undefined, undefined, Infinity), +]); + +// The same searches and windows on Buffers the compiler types statically. +const typed = Buffer.from([0x61, 0x00, 0x62, 0x00, 0x00, 0x62, 0x00, 0x63]); +const typedUnits = Buffer.from("abcabc", "utf16le"); +console.log( + "typed utf16le", + typed.indexOf(Buffer.from([0x00, 0x62]), 0, "ucs2"), + Buffer.from([0x62]).indexOf(Buffer.from([0x62]), 0, "ucs2"), + typed.lastIndexOf(Buffer.from([0x62, 0x00]), 5, "ucs2"), + typed.indexOf(Buffer.from([0x62, 0x00]), 3, "ucs2"), + typed.includes(Buffer.from([0x00, 0x62]), 0, "utf16le"), + typed.indexOf("b", 0, "ucs2"), + typedUnits.lastIndexOf("b", 100, "ucs2"), + typedUnits.lastIndexOf("b", 3, "ucs2"), + typedUnits.indexOf("", 30, "ucs2"), + typedUnits.indexOf("b", 1e300, "ucs2"), + typedUnits.indexOf("c", -1e300, "ucs2"), +); +const typedWord = Buffer.from("hello"); +console.log( + "typed windows", + JSON.stringify([ + typedWord.toString("utf8", 1, Infinity), + typedWord.toString("utf8", -Infinity, 2), + typedWord.toString("utf8", Infinity), + typedWord.toString("utf8", NaN, 3), + typedWord.toString("utf8", 1, 1e300), + typedWord.toString("hex", 0, -Infinity), + ]), +); diff --git a/tests/harness/corpus-costs/6930-dyn-buffer-methods.cjs.json b/tests/harness/corpus-costs/6930-dyn-buffer-methods.cjs.json index 83181fc36..bd8f85745 100644 --- a/tests/harness/corpus-costs/6930-dyn-buffer-methods.cjs.json +++ b/tests/harness/corpus-costs/6930-dyn-buffer-methods.cjs.json @@ -1 +1 @@ -{ "plain": 7800, "san": 39000 } +{ "plain": 12000, "san": 39000 } From e9c116859e426d85e4e1015d7e6439f659c84a1b Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 10:02:36 -0500 Subject: [PATCH 31/51] Read AST node words through a Uint32Array AstWireFile read every node field with DataView.getUint32 after checking the index with Number.isInteger and a range compare. In the compiled compiler the DataView read, Number.isInteger and its trunc are runtime calls on every access, and parent, kind, next and data run on every child of every walk. The node table is now a Uint32Array: a view of the response when the table starts on a 4-byte boundary on a little-endian host, otherwise a copy decoded once with astU32. Each accessor checks its index with index >>> 0 === index and the node count, which accepts the same indices as before, and throws the same error through an out-of-line function. Decoded strings are cached in an array indexed by string slot instead of a Map. The checks of sibling, parent, list and named-child links are unchanged and still run when each node is read. --- .../compiler/src/frontend/ts7/ast-wire.ts | 81 ++++++++++++++----- 1 file changed, 60 insertions(+), 21 deletions(-) diff --git a/packages/compiler/src/frontend/ts7/ast-wire.ts b/packages/compiler/src/frontend/ts7/ast-wire.ts index 4500ddee4..a2466a743 100644 --- a/packages/compiler/src/frontend/ts7/ast-wire.ts +++ b/packages/compiler/src/frontend/ts7/ast-wire.ts @@ -72,6 +72,33 @@ export function parseAstNodeHandle(handle: string): AstNodeHandle { }; } +/** Node fields as word offsets into a node's NODE_LEN bytes. */ +const NODE_WORDS = NODE_LEN / 4; +const WORD_KIND = NODE_OFFSET_KIND / 4; +const WORD_POS = NODE_OFFSET_POS / 4; +const WORD_END = NODE_OFFSET_END / 4; +const WORD_NEXT = NODE_OFFSET_NEXT / 4; +const WORD_PARENT = NODE_OFFSET_PARENT / 4; +const WORD_DATA = NODE_OFFSET_DATA / 4; +const WORD_FLAGS = NODE_OFFSET_FLAGS / 4; +if ( + !Number.isInteger(NODE_WORDS) || + ![WORD_KIND, WORD_POS, WORD_END, WORD_NEXT, WORD_PARENT, WORD_DATA, WORD_FLAGS].every((word) => + Number.isInteger(word), + ) +) + throw new AstDecodeError("node fields are not word-aligned"); + +/** Kept out of line, so the index check that calls it stays small enough + * to inline into every node accessor. */ +function invalidNodeIndex(index: number): never { + throw new AstDecodeError(`invalid node index ${index}`); +} + +/** Whether this host stores a Uint32Array's words little-endian, the wire's + * byte order. */ +const LITTLE_ENDIAN = new Uint8Array(new Uint32Array([1]).buffer)[0] === 1; + function hexWord(value: number): string { const digits = "0123456789abcdef"; let result = ""; @@ -93,12 +120,14 @@ export class AstWireFile { private readonly extended: number; private readonly structured: number; private readonly nodes: number; - private readonly view: DataView; - private readonly stringCache = new Map(); + /** The node table as 32-bit words: a view of the response when it is + * word-aligned on a little-endian host, otherwise a decoded copy. */ + private readonly words: Uint32Array; + /** Decoded strings by string word index / 2. */ + private readonly stringCache: (string | undefined)[]; constructor(private readonly bytes: Uint8Array) { astBounds(bytes, 0, HEADER_SIZE); - this.view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); const metadata = astU32(bytes, HEADER_OFFSET_METADATA); if (metadata >>> 24 !== PROTOCOL_VERSION) throw new AstDecodeError(`unsupported protocol version ${metadata >>> 24}`); @@ -122,6 +151,15 @@ export class AstWireFile { } this.nodeCount = (bytes.length - this.nodes) / NODE_LEN; this.stringCount = (this.strings - this.stringOffsets) / 8; + const wordCount = this.nodeCount * NODE_WORDS; + const start = bytes.byteOffset + this.nodes; + if (LITTLE_ENDIAN && start % 4 === 0) { + this.words = new Uint32Array(bytes.buffer, start, wordCount); + } else { + this.words = new Uint32Array(wordCount); + for (let i = 0; i < wordCount; i++) this.words[i] = astU32(bytes, this.nodes + i * 4); + } + this.stringCache = new Array(this.stringCount).fill(undefined); if (this.nodeCount < 2) throw new AstDecodeError("missing root"); if (this.kind(1) === KIND_NODE_LIST || this.parent(1) > 1) throw new AstDecodeError("invalid root node"); @@ -142,40 +180,41 @@ export class AstWireFile { } } - private nodeOffset(index: number): number { - if (!Number.isInteger(index) || index < 0 || index >= this.nodeCount) - throw new AstDecodeError(`invalid node index ${index}`); - return this.nodes + index * NODE_LEN; + /** The first word of node `index`. `index >>> 0 === index` accepts exactly + * the integers in [0, 2^32), so with the bound it accepts the same indices + * as an integer and range check. */ + private nodeWord(index: number): number { + if (index >>> 0 !== index || index >= this.nodeCount) invalidNodeIndex(index); + return index * NODE_WORDS; } // The constructor validates the complete fixed-width node table, and - // nodeOffset validates its index. Read each word in one operation while - // preserving little-endian decoding even in an unaligned byte view. + // nodeWord validates its index. kind(index: number): number { - return this.view.getUint32(this.nodeOffset(index) + NODE_OFFSET_KIND, true); + return this.words[this.nodeWord(index) + WORD_KIND]!; } pos(index: number): number { - return this.view.getInt32(this.nodeOffset(index) + NODE_OFFSET_POS, true); + return this.words[this.nodeWord(index) + WORD_POS]! | 0; } end(index: number): number { - return this.view.getInt32(this.nodeOffset(index) + NODE_OFFSET_END, true); + return this.words[this.nodeWord(index) + WORD_END]! | 0; } flags(index: number): number { - return this.view.getUint32(this.nodeOffset(index) + NODE_OFFSET_FLAGS, true); + return this.words[this.nodeWord(index) + WORD_FLAGS]!; } data(index: number): number { - return this.view.getUint32(this.nodeOffset(index) + NODE_OFFSET_DATA, true); + return this.words[this.nodeWord(index) + WORD_DATA]!; } next(index: number): number { - const next = this.view.getUint32(this.nodeOffset(index) + NODE_OFFSET_NEXT, true); + const next = this.words[this.nodeWord(index) + WORD_NEXT]!; if (next !== 0 && (next <= index || next >= this.nodeCount)) throw new AstDecodeError("invalid sibling link"); return next; } parent(index: number): number { - const parent = this.view.getUint32(this.nodeOffset(index) + NODE_OFFSET_PARENT, true); + const parent = this.words[this.nodeWord(index) + WORD_PARENT]!; if (parent >= this.nodeCount || (index <= 1 ? parent > index : parent >= index)) throw new AstDecodeError("invalid parent link"); return parent; @@ -188,14 +227,14 @@ export class AstWireFile { } string(index: number): string { - if (!Number.isInteger(index) || index < 0 || index % 2 !== 0 || index >= this.stringCount * 2) + if (index >>> 0 !== index || index % 2 !== 0 || index >= this.stringCount * 2) throw new AstDecodeError(`invalid string index ${index}`); - const cached = this.stringCache.get(index); + const cached = this.stringCache[index / 2]; if (cached !== undefined) return cached; const start = astU32(this.bytes, this.stringOffsets + index * 4); const end = astU32(this.bytes, this.stringOffsets + (index + 1) * 4); const text = decodeAstString(this.bytes, this.strings + start, end - start); - this.stringCache.set(index, text); + this.stringCache[index / 2] = text; return text; } @@ -250,7 +289,7 @@ export class AstWireFile { } firstChild(index: number): number { - this.nodeOffset(index); + this.nodeWord(index); return index + 1 < this.nodeCount && this.parent(index + 1) === index ? index + 1 : 0; } @@ -360,7 +399,7 @@ export class AstWireFile { const result: number[] = []; for (let i = 0; i < count; i++) { const index = reader.uint(); - this.nodeOffset(index); + this.nodeWord(index); if (index === 0 || this.kind(index) === KIND_NODE_LIST) throw new AstDecodeError("structured reference is not a node"); result.push(index); From f793009ee5cabf4db5da38b9c78ebfc96a8de4fd Mon Sep 17 00:00:00 2001 From: Malte Ubl Date: Tue, 20 Oct 2026 13:34:29 -0700 Subject: [PATCH 32/51] Unwrap tsc's interop helpers around builtin modules in packages tsc wraps every CommonJS import in its helpers: `const events_1 = __importDefault(require("events"))`, `const path = __importStar(require("path"))`. cjs-tsc-interop.ts reads such bindings of program modules as plain require bindings, so the helper declarations go dead, but a builtin module is not a program module: its binding kept the helpers live, and their probe of the module's top-level `this` (`(this && this.__importDefault) ||`) failed at load ("property access through top-level 'this'"). That is the shape of hundreds of tsc-compiled files in a typical dependency tree. A builtin carries no `__esModule`, so `events_1.default` is the module and the namespace copy reads the module's members. When every reference has that form, a package-file stage now respells the binding as the plain require() of the builtin and `x.default` as `x`. Co-Authored-By: Claude Opus 5.5 --- .../frontend/npm-static-tsc-helpers.test.ts | 71 ++++++++++++ .../src/frontend/npm-static-tsc-helpers.ts | 106 ++++++++++++++++++ packages/compiler/src/frontend/npm-static.ts | 4 + packages/compiler/src/frontend/services.ts | 8 ++ .../cases/tsc-builtin-interop/main.ts.json | 6 + .../npm/cases/tsc-builtin-interop/main.ts | 9 ++ .../node_modules/gttscinterop/lib/helper.d.ts | 1 + .../node_modules/gttscinterop/lib/helper.js | 6 + .../node_modules/gttscinterop/lib/index.d.ts | 5 + .../node_modules/gttscinterop/lib/index.js | 43 +++++++ .../node_modules/gttscinterop/package.json | 7 ++ 11 files changed, 266 insertions(+) create mode 100644 packages/compiler/src/frontend/npm-static-tsc-helpers.test.ts create mode 100644 packages/compiler/src/frontend/npm-static-tsc-helpers.ts create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/fixtures/npm/cases/tsc-builtin-interop/main.ts.json create mode 100644 tests/fixtures/npm/cases/tsc-builtin-interop/main.ts create mode 100644 tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/helper.d.ts create mode 100644 tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/helper.js create mode 100644 tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/index.d.ts create mode 100644 tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/index.js create mode 100644 tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/package.json diff --git a/packages/compiler/src/frontend/npm-static-tsc-helpers.test.ts b/packages/compiler/src/frontend/npm-static-tsc-helpers.test.ts new file mode 100644 index 000000000..735c9d298 --- /dev/null +++ b/packages/compiler/src/frontend/npm-static-tsc-helpers.test.ts @@ -0,0 +1,71 @@ +import { spawnSync } from "node:child_process"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterAll, expect, test } from "vitest"; +import { nodeFrontendServices } from "./services-node.js"; + +const dirs: string[] = []; +afterAll(() => { + for (const dir of dirs) rmSync(dir, { recursive: true, force: true }); +}); + +function rewrite(source: string): string | null { + return nodeFrontendServices().tscBuiltinImports(source, "/pkg/index.js"); +} + +const helpers = `"use strict"; +var __importDefault = (this && this.__importDefault) || function (mod) { + return (mod && mod.__esModule) ? mod : { "default": mod }; +}; +var __importStar = (this && this.__importStar) || function (mod) { + if (mod && mod.__esModule) return mod; + var result = {}; + if (mod != null) for (var k in mod) if (k !== "default" && Object.prototype.hasOwnProperty.call(mod, k)) result[k] = mod[k]; + result["default"] = mod; + return result; +}; +`; + +function expectSameBehavior(source: string): string { + const result = rewrite(source); + expect(typeof result).toBe("string"); + const dir = mkdtempSync(join(tmpdir(), "scriptc-tsc-builtin-imports-")); + dirs.push(dir); + writeFileSync(join(dir, "original.cjs"), source); + writeFileSync(join(dir, "rewritten.cjs"), result!); + const run = (file: string) => + spawnSync(process.execPath, [join(dir, file)], { encoding: "utf8" }); + const oracle = run("original.cjs"); + expect(oracle.status).toBe(0); + expect(run("rewritten.cjs").stdout).toBe(oracle.stdout); + return result!; +} + +test("builtin modules behind tsc's interop helpers become plain requires", () => { + const out = expectSameBehavior(`${helpers}const events_1 = __importDefault(require("events")); +const path = __importStar(require("node:path")); +class Bus extends events_1.default.EventEmitter {} +console.log(new Bus() instanceof events_1.default, path.basename("/a/b.txt"), typeof path.join); +`); + expect(out).toContain('const events_1 = require("events");'); + expect(out).toContain("class Bus extends events_1.EventEmitter {}"); + expect(out).toContain('const path = require("node:path");'); +}); + +test("other references, program modules and default reads of a namespace keep their text", () => { + // The binding itself as a value. + expect( + rewrite( + `${helpers}const events_1 = __importDefault(require("events"));\nconsole.log(events_1);\n`, + ), + ).toBeNull(); + // A program module (cjs-tsc-interop.ts owns it). + expect( + rewrite(`${helpers}const m = __importDefault(require("./m"));\nm.default();\n`), + ).toBeNull(); + // `ns.default` reads the namespace copy's own default. + expect( + rewrite(`${helpers}const p = __importStar(require("path"));\nconsole.log(p.default === p);\n`), + ).toBeNull(); +}); diff --git a/packages/compiler/src/frontend/npm-static-tsc-helpers.ts b/packages/compiler/src/frontend/npm-static-tsc-helpers.ts new file mode 100644 index 000000000..fcb73be5b --- /dev/null +++ b/packages/compiler/src/frontend/npm-static-tsc-helpers.ts @@ -0,0 +1,106 @@ +/* tsc wraps every CommonJS import in its interop helpers: + * + * const events_1 = __importDefault(require("events")); // events_1.default.X + * const path = __importStar(require("path")); // path.basename + * + * For program modules cjs-tsc-interop.ts reads such a binding as a plain + * require binding, and the helper declarations (`var __importDefault = + * (this && this.__importDefault) || …`) go dead. A builtin module is not a + * program module, so its binding kept the helpers live, and their probe of + * the module's top-level `this` has no lowering. A builtin carries no + * `__esModule`: `events_1.default` is the module and the namespace copy + * reads the module's members. When every reference has that form + * (`x.default…` for the default helper, `x.` other than `default` + * for the namespace helper), the binding becomes the plain `require()` of + * the builtin and `x.default` becomes `x`. */ +import * as ts from "./ts7/syntax.js"; +import { applyEdits, unparen, type Edit } from "./npm-static-bundled-chunks.js"; +import { canonicalBuiltinModule } from "./builtin-modules.js"; + +/** True when the text may hold such a binding (a cheap probe). */ +export function mayHoldTscBuiltinImport(source: string): boolean { + return /__import(?:Default|Star)\(require\(/.test(source); +} + +/** `__importDefault(require(""))` / `__importStar(…)`: the helper + * and builtin spec, or null. */ +function builtinInterop(init: ts.Expression): { helper: string; spec: string } | null { + const e = unparen(init); + if (!ts.isCallExpression(e) || !ts.isIdentifier(e.expression) || e.arguments.nodes.length !== 1) + return null; + const helper = e.expression.text; + if (helper !== "__importDefault" && helper !== "__importStar") return null; + const req = unparen(e.arguments.nodes[0]!); + if ( + !ts.isCallExpression(req) || + !ts.isIdentifier(req.expression) || + req.expression.text !== "require" || + req.arguments.nodes.length !== 1 || + !ts.isStringLiteral(req.arguments.nodes[0]!) + ) + return null; + const spec = (req.arguments.nodes[0] as ts.StringLiteral).text; + return canonicalBuiltinModule(spec) === null ? null : { helper, spec }; +} + +/** The `.default` accesses to remove when every reference to `name` (other + * than its declaration) has the helper's form, or null. */ +function interopReferences( + sf: ts.SourceFile, + decl: ts.VariableDeclaration, + name: string, + helper: string, +): ts.PropertyAccessExpression[] | null { + const defaults: ts.PropertyAccessExpression[] = []; + let ok = true; + ts.walkPreorder(sf, (n) => { + if (!ok) return "stop"; + if (!ts.isIdentifier(n) || n.text !== name || n === decl.name) return undefined; + const parent = n.parent; + if ( + parent === undefined || + !ts.isPropertyAccessExpression(parent) || + parent.expression !== n || + parent.questionDotToken !== undefined + ) { + ok = false; + return "stop"; + } + const member = parent.name.text; + if (helper === "__importDefault") { + if (member !== "default") ok = false; + else defaults.push(parent); + } else if (member === "default") { + ok = false; + } + return undefined; + }); + return ok ? defaults : null; +} + +/** The module text with its builtin interop bindings unwrapped, or null. */ +export function unwrapTscBuiltinImports(sf: ts.SourceFile): string | null { + if (ts.isExternalModule(sf) && !sf.fileName.endsWith(".cjs")) return null; + const edits: Edit[] = []; + const stmts = sf.statements.nodes; + // Builtin modules through the interop helpers. + for (const stmt of stmts) { + if (!ts.isVariableStatement(stmt)) continue; + for (const d of stmt.declarationList.declarations.nodes) { + if (!ts.isIdentifier(d.name) || d.initializer === undefined) continue; + const interop = builtinInterop(d.initializer); + if (interop === null) continue; + const defaults = interopReferences(sf, d, d.name.text, interop.helper); + if (defaults === null) continue; + edits.push({ + start: d.initializer.getStart(sf), + end: d.initializer.getEnd(), + text: `require(${JSON.stringify(interop.spec)})`, + }); + for (const access of defaults) + edits.push({ start: access.getStart(sf), end: access.getEnd(), text: d.name.text }); + } + } + if (edits.length === 0) return null; + return applyEdits(sf.text, 0, sf.text.length, edits); +} diff --git a/packages/compiler/src/frontend/npm-static.ts b/packages/compiler/src/frontend/npm-static.ts index 1bf84e734..bc9fc2abd 100644 --- a/packages/compiler/src/frontend/npm-static.ts +++ b/packages/compiler/src/frontend/npm-static.ts @@ -455,6 +455,9 @@ export function npmStaticFsShadow(services: FrontendServices): NpmStaticFsShadow // Polyfill packages that ARE a Node builtin compile as the builtin. const aliased = original === null ? null : services.builtinAliasRequires(original, path); if (aliased !== null) original = aliased; + // tsc's interop helpers around builtin modules. + const tscHelpers = original === null ? null : services.tscBuiltinImports(original, path); + if (tscHelpers !== null) original = tscHelpers; // A module exporting one of its functions reaches its members // through that function. const fnExports = original === null ? null : services.cjsFunctionExports(original, path); @@ -469,6 +472,7 @@ export function npmStaticFsShadow(services: FrontendServices): NpmStaticFsShadow if (exportsVar !== null) original = exportsVar; const respelled = split !== null || + tscHelpers !== null || esm !== null || aliased !== null || fnExports !== null || diff --git a/packages/compiler/src/frontend/services.ts b/packages/compiler/src/frontend/services.ts index be3881e13..10931ab29 100644 --- a/packages/compiler/src/frontend/services.ts +++ b/packages/compiler/src/frontend/services.ts @@ -13,6 +13,7 @@ import { rewriteBuiltinAliasRequires, } from "./npm-static-builtin-aliases.js"; import { mayHoldEnumIife, rewriteEnumIifes } from "./npm-static-enum-iife.js"; +import { mayHoldTscBuiltinImport, unwrapTscBuiltinImports } from "./npm-static-tsc-helpers.js"; import { builtinPackageAliasFor } from "./package-shims.js"; import * as syntax from "./npm-static-declaration-syntax.js"; import { normalizeCjsExportsVar } from "./npm-static-cjs-exports-var.js"; @@ -106,6 +107,13 @@ export class FrontendServices { if (!source.includes("module.exports")) return null; return normalizeCjsFunctionExports(this.parse(path, source, "js")); } + /** A package module with tsc's interop bindings of builtin modules + * unwrapped to plain requires (null: none). */ + tscBuiltinImports(source: string, path: string): string | null { + this.ensureOpen(); + if (!mayHoldTscBuiltinImport(source)) return null; + return unwrapTscBuiltinImports(this.parse(path, source, "js")); + } /** A package module with its literal-valued TypeScript enum IIFEs * respelled as object literals (null: none). */ enumIifes(source: string, path: string): string | null { diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/fixtures/npm/cases/tsc-builtin-interop/main.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/fixtures/npm/cases/tsc-builtin-interop/main.ts.json new file mode 100644 index 000000000..cdb86be04 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/fixtures/npm/cases/tsc-builtin-interop/main.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/fixtures/npm/cases/tsc-builtin-interop/main.ts" + ], + "diags": [] +} diff --git a/tests/fixtures/npm/cases/tsc-builtin-interop/main.ts b/tests/fixtures/npm/cases/tsc-builtin-interop/main.ts new file mode 100644 index 000000000..99bbe020a --- /dev/null +++ b/tests/fixtures/npm/cases/tsc-builtin-interop/main.ts @@ -0,0 +1,9 @@ +// A tsc-compiled dependency wraps builtin modules in its interop helpers +// (`const events_1 = __importDefault(require("events"))`, then +// `class Bus extends events_1.default.EventEmitter`; `const path = +// __importStar(require("path"))`). +import { Bus, where } from "gttscinterop"; + +const bus = new Bus(); +bus.on("ping", (name: string) => console.log("got", name)); +console.log(bus.ping("a"), where("/x/y.txt")); diff --git a/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/helper.d.ts b/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/helper.d.ts new file mode 100644 index 000000000..cebef9119 --- /dev/null +++ b/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/helper.d.ts @@ -0,0 +1 @@ +export default function suffix(): string; diff --git a/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/helper.js b/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/helper.js new file mode 100644 index 000000000..9c847c6ac --- /dev/null +++ b/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/helper.js @@ -0,0 +1,6 @@ +"use strict"; +Object.defineProperty(exports, "__esModule", { value: true }); +exports.default = suffix; +function suffix() { + return "!"; +} diff --git a/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/index.d.ts b/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/index.d.ts new file mode 100644 index 000000000..f185ed258 --- /dev/null +++ b/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/index.d.ts @@ -0,0 +1,5 @@ +import events from "events"; +export declare class Bus extends events.EventEmitter { + ping(name: string): string; +} +export declare function where(p: string): string; diff --git a/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/index.js b/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/index.js new file mode 100644 index 000000000..96450e0e5 --- /dev/null +++ b/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/lib/index.js @@ -0,0 +1,43 @@ +"use strict"; +var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) { + if (k2 === undefined) k2 = k; + var desc = Object.getOwnPropertyDescriptor(m, k); + if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) { + desc = { enumerable: true, get: function() { return m[k]; } }; + } + Object.defineProperty(o, k2, desc); +}) : (function(o, m, k, k2) { + if (k2 === undefined) k2 = k; + o[k2] = m[k]; +})); +var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) { + Object.defineProperty(o, "default", { enumerable: true, value: v }); +}) : function(o, v) { + o["default"] = v; +}); +var __importStar = (this && this.__importStar) || function (mod) { + if (mod && mod.__esModule) return mod; + var result = {}; + if (mod != null) for (var k in mod) if (k !== "default" && Object.prototype.hasOwnProperty.call(mod, k)) __createBinding(result, mod, k); + __setModuleDefault(result, mod); + return result; +}; +var __importDefault = (this && this.__importDefault) || function (mod) { + return (mod && mod.__esModule) ? mod : { "default": mod }; +}; +Object.defineProperty(exports, "__esModule", { value: true }); +exports.Bus = void 0; +exports.where = where; +const events_1 = __importDefault(require("events")); +const path = __importStar(require("path")); +const helper_1 = __importDefault(require("./helper")); +class Bus extends events_1.default.EventEmitter { + ping(name) { + this.emit("ping", name); + return `pong ${name}${(0, helper_1.default)()}`; + } +} +exports.Bus = Bus; +function where(p) { + return path.basename(p) + (0, helper_1.default)(); +} diff --git a/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/package.json b/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/package.json new file mode 100644 index 000000000..bc83710c8 --- /dev/null +++ b/tests/fixtures/npm/cases/tsc-builtin-interop/node_modules/gttscinterop/package.json @@ -0,0 +1,7 @@ +{ + "name": "gttscinterop", + "version": "1.0.0", + "description": "real tsc 5.6 output (module commonjs, esModuleInterop) of a source that default-imports `events` and namespace-imports `path`: the builtins are wrapped in tsc's __importDefault/__importStar helpers, which probe the module's top-level `this`", + "main": "lib/index.js", + "types": "lib/index.d.ts" +} From 858328703bdd42badc3c1abd1eef6f016b9a4207 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 10:34:21 -0500 Subject: [PATCH 33/51] Name typed functions' property tables and keep runtime freezes data-only The first Object.freeze, Object.seal or Object.preventExtensions of a typed function creates its property table from the function's dynamic box. The box had no name, so later `f.name` reads answered "". The box now carries the function's JS name, as a closure flowing into any checked-dynamic slot does. The runtime's own callers (published values, os and node constants, the private-range block list, Wasm exports) freeze data they built, and now call scr_dyn_freeze_data, the freeze for values that are not functions. Object.isFrozen, Object.isSealed and Object.isExtensible read a function's existing table and answer false, false and true when it has none. Fully typed programs therefore link no dynamic-tier object code for these calls again. --- .../src/frontend/lowering/lower-calls.ts | 23 ++++++-- .../corpus/function-integrity-names.ts.json | 6 ++ packages/runtime/src/scr_dyn.h | 5 ++ packages/runtime/src/scr_json.c | 58 ++++++++++++------- packages/runtime/src/scr_net.c | 2 +- packages/runtime/src/scr_os.c | 4 +- packages/runtime/src/scr_publish.c | 2 +- packages/runtime/src/scr_wasm.c | 2 +- tests/corpus/function-integrity-names.ts | 21 +++++++ tests/corpus/function-integrity.js | 3 +- .../function-integrity-names.ts.json | 1 + .../corpus-costs/function-integrity.js.json | 1 + 12 files changed, 98 insertions(+), 30 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/function-integrity-names.ts.json create mode 100644 tests/corpus/function-integrity-names.ts create mode 100644 tests/harness/corpus-costs/function-integrity-names.ts.json create mode 100644 tests/harness/corpus-costs/function-integrity.js.json diff --git a/packages/compiler/src/frontend/lowering/lower-calls.ts b/packages/compiler/src/frontend/lowering/lower-calls.ts index 581cf0f5c..0823987c6 100644 --- a/packages/compiler/src/frontend/lowering/lower-calls.ts +++ b/packages/compiler/src/frontend/lowering/lower-calls.ts @@ -8071,7 +8071,13 @@ function lowerObjectStaticCall( loc, }; } - const fnIntegrity = functionIntegrity(lowerer, member, value, locOf(call)); + const fnIntegrity = functionIntegrity( + lowerer, + member, + value, + call.arguments.nodes[0]!, + locOf(call), + ); if (fnIntegrity) return fnIntegrity; if (member === "preventExtensions" || member === "seal") { const integrity = lowerClassIntegrity(lowerer, call, member, value); @@ -8156,7 +8162,7 @@ function lowerObjectStaticCall( ) ) return value; - const fnIntegrity = functionIntegrity(lowerer, "freeze", value, locOf(call)); + const fnIntegrity = functionIntegrity(lowerer, "freeze", value, argNode, locOf(call)); if (fnIntegrity) return fnIntegrity; const integrity = lowerClassIntegrity(lowerer, call, "freeze", value); if (integrity) return integrity; @@ -10654,6 +10660,7 @@ function functionIntegrity( lowerer: Lowerer, member: string, value: IrExpr, + node: ts.Expression, loc: SrcLoc, ): IrExpr | null { if (value.type.kind !== "func" || !lowerer.dynConvertible(value.type)) return null; @@ -10666,8 +10673,16 @@ function functionIntegrity( } as const; if (!(member in operations)) return null; const fn = operations[member as keyof typeof operations]; + // The first operation on a function creates its property table, whose + // `name` comes from the box: give the box the function's JS name, as a + // closure flowing into any checked-dynamic slot gets. + const name = jsFuncNameOf(node, lowerer); + const boxed = (e: IrExpr): IrExpr => { + const box = lowerer.coerceToExpected(e, DYN); + return box.kind === "dynFrom" && name !== null ? { ...box, fnName: name } : box; + }; if (fn === "dyn.isSealed" || fn === "dyn.isExtensible") { - return { kind: "libCall", fn, args: [lowerer.coerceToExpected(value, DYN)], type: BOOL, loc }; + return { kind: "libCall", fn, args: [boxed(value)], type: BOOL, loc }; } const held = lowerer.declareHiddenLocal("%integrityFn", value.type); const ref = (): IrExpr => ({ kind: "varRef", localId: held.id, type: value.type, loc }); @@ -10677,7 +10692,7 @@ function functionIntegrity( { kind: "varDecl", localId: held.id, init: value, loc }, { kind: "exprStmt", - expr: { kind: "libCall", fn, args: [lowerer.coerceToExpected(ref(), DYN)], type: DYN, loc }, + expr: { kind: "libCall", fn, args: [boxed(ref())], type: DYN, loc }, loc, }, ], diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/function-integrity-names.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/function-integrity-names.ts.json new file mode 100644 index 000000000..f3a79fd5b --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/function-integrity-names.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/function-integrity-names.ts" + ], + "diags": [] +} diff --git a/packages/runtime/src/scr_dyn.h b/packages/runtime/src/scr_dyn.h index a31f7c49e..addeb2cdc 100644 --- a/packages/runtime/src/scr_dyn.h +++ b/packages/runtime/src/scr_dyn.h @@ -671,6 +671,11 @@ double scr_dyn_native_date_value(const ScrDyn *value); ScrDyn *scr_dyn_native_regex(ScrRegex *value); bool scr_dyn_native_regex_is(const ScrDyn *value); ScrDyn *scr_dyn_freeze(ScrDyn *value); +/* scr_dyn_freeze of a value that is not a function (objects, arrays, + * primitives), for runtime callers freezing data they built. It never + * creates a function's property table, so it adds no dynamic-tier code to + * the programs that link it. */ +ScrDyn *scr_dyn_freeze_data(ScrDyn *value); bool scr_dyn_is_frozen(const ScrDyn *value); ScrDyn *scr_dyn_new_bigint(ScrBigInt *value); /* borrows and retains value */ ScrDyn *scr_dyn_new_symbol(ScrSym *value); /* borrows and retains value */ diff --git a/packages/runtime/src/scr_json.c b/packages/runtime/src/scr_json.c index f19bc7083..117b733ee 100644 --- a/packages/runtime/src/scr_json.c +++ b/packages/runtime/src/scr_json.c @@ -1993,11 +1993,24 @@ static bool scr_dyn_is_extensible_view(const ScrDyn *view) { /* A plain function's own properties (name, length, prototype, and the ones * a program adds) live in its property table, which carries the function's * integrity state too: freezing or sealing the function freezes or seals - * the table. +1, or NULL for other values and for class constructors, whose - * statics keep their own fences. */ + * the table. Class constructors are not plain functions: their statics keep + * their own fences. */ +static bool scr_dyn_plain_fn(const ScrDyn *object) { + return object->kind == SCR_DYN_FUNC && !object->v.fn.class_obj; +} + +/* The table of a plain function for an integrity operation, created on + * first use. +1, or NULL for other values. */ static ScrDyn *scr_dyn_fn_integrity_table(const ScrDyn *object) { - if (object->kind != SCR_DYN_FUNC || object->v.fn.class_obj) return NULL; - return scr_dyn_fn_properties(object); + return scr_dyn_plain_fn(object) ? scr_dyn_fn_properties(object) : NULL; +} + +/* The table a plain function already has, for the integrity queries: one + * without a table has never been frozen, sealed or made non-extensible. + * +1, or NULL. */ +static ScrDyn *scr_dyn_fn_existing_table(const ScrDyn *function) { + ScrClosure *owner = scr_closure_identity(function->v.fn.clo); + return owner->props ? (ScrDyn *)scr_box_get_ref(owner->props) : NULL; } ScrDyn *scr_dyn_prevent_extensions(ScrDyn *object) { @@ -2022,9 +2035,9 @@ bool scr_dyn_is_extensible(ScrDyn *object) { if (object->kind == SCR_DYN_PROXY) { scr_dyn_proxy_unsupported("Object.isExtensible"); return false; } bool answer; if (scr_dyn_typed_ref_view_answer(object, scr_dyn_is_extensible_view, &answer)) return answer; - ScrDyn *table = scr_dyn_fn_integrity_table(object); - if (table) { - answer = !table->non_extensible; + if (scr_dyn_plain_fn(object)) { + ScrDyn *table = scr_dyn_fn_existing_table(object); + answer = !table || !table->non_extensible; scr_dyn_release(table); return answer; } @@ -2059,9 +2072,9 @@ bool scr_dyn_is_sealed(const ScrDyn *object) { bool answer; if (scr_dyn_typed_ref_view_answer((ScrDyn *)object, scr_dyn_is_sealed, &answer)) return answer; if (object->symbol_properties && !scr_dyn_is_sealed(object->symbol_properties)) return false; - ScrDyn *table = scr_dyn_fn_integrity_table(object); - if (table) { - answer = scr_dyn_is_sealed(table); + if (scr_dyn_plain_fn(object)) { + ScrDyn *table = scr_dyn_fn_existing_table(object); + answer = table && scr_dyn_is_sealed(table); scr_dyn_release(table); return answer; } @@ -14414,18 +14427,23 @@ ScrStr *scr_dyn_to_string_method(const ScrDyn *d, const ScrStr *enc, const ScrSt ScrDyn *scr_dyn_freeze(ScrDyn *value) { if (scr_dyn_class_reflection_fence(value)) return NULL; if (scr_dyn_class_integrity_fence(value, "Object.freeze")) return NULL; - if (value->symbol_properties) scr_dyn_release(scr_dyn_freeze(value->symbol_properties)); ScrDyn *table = scr_dyn_fn_integrity_table(value); - if (table) { - scr_dyn_release(scr_dyn_freeze(table)); - scr_dyn_release(table); - return scr_dyn_retain(value); - } + if (!table) return scr_dyn_freeze_data(value); + if (value->symbol_properties) scr_dyn_release(scr_dyn_freeze_data(value->symbol_properties)); + scr_dyn_release(scr_dyn_freeze_data(table)); + scr_dyn_release(table); + return scr_dyn_retain(value); +} + +ScrDyn *scr_dyn_freeze_data(ScrDyn *value) { + if (scr_dyn_class_reflection_fence(value)) return NULL; + if (scr_dyn_class_integrity_fence(value, "Object.freeze")) return NULL; + if (value->symbol_properties) scr_dyn_release(scr_dyn_freeze_data(value->symbol_properties)); if (value->kind == SCR_DYN_ARR) { value->non_extensible = true; value->v.arr.sealed = true; value->v.arr.frozen = true; - if (value->v.arr.properties) scr_dyn_release(scr_dyn_freeze(value->v.arr.properties)); + if (value->v.arr.properties) scr_dyn_release(scr_dyn_freeze_data(value->v.arr.properties)); return scr_dyn_retain(value); } if (value->kind == SCR_DYN_OBJ && !value->copied_from_native) { @@ -14450,9 +14468,9 @@ bool scr_dyn_is_frozen(const ScrDyn *value) { bool answer; if (scr_dyn_typed_ref_view_answer((ScrDyn *)value, scr_dyn_is_frozen, &answer)) return answer; if (value->symbol_properties && !scr_dyn_is_frozen(value->symbol_properties)) return false; - ScrDyn *table = scr_dyn_fn_integrity_table(value); - if (table) { - answer = scr_dyn_is_frozen(table); + if (scr_dyn_plain_fn(value)) { + ScrDyn *table = scr_dyn_fn_existing_table(value); + answer = table && scr_dyn_is_frozen(table); scr_dyn_release(table); return answer; } diff --git a/packages/runtime/src/scr_net.c b/packages/runtime/src/scr_net.c index 765b59a05..ab7da1ebd 100644 --- a/packages/runtime/src/scr_net.c +++ b/packages/runtime/src/scr_net.c @@ -4329,7 +4329,7 @@ ScrDyn *scr_net_block_list_private_ranges(void) { "::1/128", "169.254.0.0/16", "fe80::/10", "fc00::/7" }; ScrDyn *arr = scr_dyn_new_arr(); for (size_t i = 0; i < sizeof cidrs / sizeof cidrs[0]; i++) scr_dyn_arr_push(arr, scr_net_dyn_str(cidrs[i])); - ranges = scr_dyn_freeze(arr); /* +1 on the same array */ + ranges = scr_dyn_freeze_data(arr); /* +1 on the same array */ scr_dyn_release(arr); scr_net_private_ranges = ranges; scr_atexit(scr_net_private_ranges_cleanup); diff --git a/packages/runtime/src/scr_os.c b/packages/runtime/src/scr_os.c index 500283995..8fa68738d 100644 --- a/packages/runtime/src/scr_os.c +++ b/packages/runtime/src/scr_os.c @@ -1327,7 +1327,7 @@ ScrDyn *scr_os_constants(void) { scr_os_constant(os, "UV_UDP_REUSEADDR", 4); scr_dyn_obj_set(os, "dlopen", 6, dlopen); scr_dyn_obj_set(os, "errno", 5, errnos); - scr_dyn_release(scr_dyn_freeze(signals)); + scr_dyn_release(scr_dyn_freeze_data(signals)); scr_dyn_obj_set(os, "signals", 7, signals); scr_dyn_obj_set(os, "priority", 8, priority); scr_os_constants_value = os; @@ -1521,7 +1521,7 @@ ScrDyn *scr_node_constants(const ScrDyn *crypto_table) { } } scr_dyn_release(crypto); - scr_dyn_release(scr_dyn_freeze(o)); + scr_dyn_release(scr_dyn_freeze_data(o)); scr_node_constants_value = o; scr_atexit(scr_node_constants_clear); return scr_dyn_retain(o); diff --git a/packages/runtime/src/scr_publish.c b/packages/runtime/src/scr_publish.c index 0cbb02c8f..7a15f0a61 100644 --- a/packages/runtime/src/scr_publish.c +++ b/packages/runtime/src/scr_publish.c @@ -460,7 +460,7 @@ void scr_publish(void *root, ScrPubFn fn) { /* Other threads share the object; its shape belongs to this one * (scr_js.h). */ if (d->kind == SCR_DYN_OBJ) d->v.obj.shape = NULL; - scr_dyn_release(scr_dyn_freeze(d)); + scr_dyn_release(scr_dyn_freeze_data(d)); } #ifdef SCR_RC_AUDIT scr_pub_audit_forget(ctx.kinds[i]); diff --git a/packages/runtime/src/scr_wasm.c b/packages/runtime/src/scr_wasm.c index 121acacbb..65132a941 100644 --- a/packages/runtime/src/scr_wasm.c +++ b/packages/runtime/src/scr_wasm.c @@ -2043,7 +2043,7 @@ static ScrDyn *scr_wasm_instance_from(const ScrWasmModuleDesc *desc, ScrDyn *imp } scr_dyn_obj_set(exports, ex->name, (size_t)ex->name_len, value); } - ScrDyn *frozen = scr_dyn_freeze(exports); + ScrDyn *frozen = scr_dyn_freeze_data(exports); scr_dyn_release(frozen); host->exports = exports; ScrWasmObj *o = scr_wasm_obj_new(SCR_WASM_K_INSTANCE); diff --git a/tests/corpus/function-integrity-names.ts b/tests/corpus/function-integrity-names.ts new file mode 100644 index 000000000..eb72a8691 --- /dev/null +++ b/tests/corpus/function-integrity-names.ts @@ -0,0 +1,21 @@ +// Integrity operations on typed functions keep their names: the first one +// creates the function's property table, whose `name` must be the +// function's own. Queries on functions no operation has touched answer +// without creating a table. +function hashC(x: number): number { + return x + 1; +} +const arrow = (s: string): string => s + "!"; +const untouched = (a: number, b: number): number => a * b; +console.log(Object.isFrozen(untouched), Object.isSealed(untouched), Object.isExtensible(untouched)); +Object.freeze(hashC); +Object.seal(arrow); +console.log(Object.isFrozen(hashC), Object.isSealed(hashC), Object.isExtensible(hashC), hashC(1)); +console.log(Object.isFrozen(arrow), Object.isSealed(arrow), Object.isExtensible(arrow), arrow("a")); +function later(): number { + return 7; +} +Object.preventExtensions(later); +console.log(Object.isFrozen(later), Object.isSealed(later), Object.isExtensible(later), later()); +const functions: any[] = [hashC, arrow, untouched, later]; +console.log(functions.map((f) => `${f.name}/${f.length}`).join(" ")); diff --git a/tests/corpus/function-integrity.js b/tests/corpus/function-integrity.js index 4258a7838..59a5fbfec 100644 --- a/tests/corpus/function-integrity.js +++ b/tests/corpus/function-integrity.js @@ -32,7 +32,7 @@ plain.tag = "t"; Object.seal(plain); console.log(Object.isFrozen(plain), Object.isSealed(plain), Object.isExtensible(plain)); plain.tag = "u"; -console.log(plain.tag, plain(1, 2)); +console.log(plain.tag, plain(1, 2), plain.name, plain.length); const stopped = () => 0; Object.preventExtensions(stopped); @@ -42,3 +42,4 @@ try { } catch (error) { console.log(error.name, error.message); } +console.log(stopped.name, stopped.length); diff --git a/tests/harness/corpus-costs/function-integrity-names.ts.json b/tests/harness/corpus-costs/function-integrity-names.ts.json new file mode 100644 index 000000000..9cc950b57 --- /dev/null +++ b/tests/harness/corpus-costs/function-integrity-names.ts.json @@ -0,0 +1 @@ +{ "plain": 3907, "san": 4954 } diff --git a/tests/harness/corpus-costs/function-integrity.js.json b/tests/harness/corpus-costs/function-integrity.js.json new file mode 100644 index 000000000..1970272f3 --- /dev/null +++ b/tests/harness/corpus-costs/function-integrity.js.json @@ -0,0 +1 @@ +{ "plain": 3345, "san": 3989 } From cd38fd2248fefeddcff452670bac65d07e9a129f Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 11:03:19 -0500 Subject: [PATCH 34/51] Bridge events whose typed listeners take no parameters An event that typed code only listened to with a listener taking no parameters had no bridge entry: the scan marked a listener as typed only when one of its parameters was not a checked-dynamic value. Emitting the event through an `any` reference then threw instead of calling the listener. A TypeScript listener with no parameters now marks the event as typed. A JavaScript function without parameters still registers as before, since it can read `arguments`. --- .../frontend/lowering/lower-event-emitter.ts | 8 ++++++- ...-emitter-untyped-zero-arg-listener.ts.json | 6 +++++ .../6941-emitter-untyped-zero-arg-listener.ts | 23 +++++++++++++++++++ ...-emitter-untyped-zero-arg-listener.ts.json | 1 + 4 files changed, 37 insertions(+), 1 deletion(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6941-emitter-untyped-zero-arg-listener.ts.json create mode 100644 tests/corpus/6941-emitter-untyped-zero-arg-listener.ts create mode 100644 tests/harness/corpus-costs/6941-emitter-untyped-zero-arg-listener.ts.json diff --git a/packages/compiler/src/frontend/lowering/lower-event-emitter.ts b/packages/compiler/src/frontend/lowering/lower-event-emitter.ts index 6b79666b1..d84f84f4e 100644 --- a/packages/compiler/src/frontend/lowering/lower-event-emitter.ts +++ b/packages/compiler/src/frontend/lowering/lower-event-emitter.ts @@ -338,9 +338,15 @@ function emitterEvents(lowerer: Lowerer): Map { const cbCt = lowerer.typeOf(node.arguments.nodes[1]); const cbT = lowerer.mapTypeOf(cbCt) ?? dynFallbackType(lowerer, node.arguments.nodes[1], cbCt); + // A TypeScript listener with no parameters is typed too: it + // registers without the adapter, so an untyped emit reaches it + // only through the event's fixed ABI. A JavaScript function + // without parameters can still read `arguments`. if ( cbT?.kind !== "dyn" && - (cbT?.kind !== "func" || cbT.params.some((p) => p.kind !== "dyn")) + (cbT?.kind !== "func" || + (cbT.params.length === 0 && !isJsSourceFile(sf)) || + cbT.params.some((p) => p.kind !== "dyn")) ) { sigOf(name).typedListener = true; } diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6941-emitter-untyped-zero-arg-listener.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6941-emitter-untyped-zero-arg-listener.ts.json new file mode 100644 index 000000000..5e0afea43 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6941-emitter-untyped-zero-arg-listener.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6941-emitter-untyped-zero-arg-listener.ts" + ], + "diags": [] +} diff --git a/tests/corpus/6941-emitter-untyped-zero-arg-listener.ts b/tests/corpus/6941-emitter-untyped-zero-arg-listener.ts new file mode 100644 index 000000000..89c4dc70f --- /dev/null +++ b/tests/corpus/6941-emitter-untyped-zero-arg-listener.ts @@ -0,0 +1,23 @@ +// Emitting through an `any` reference reaches listeners that typed code +// registered with no parameters, for an event no typed code emits: they +// run once per emit, whatever arguments the untyped emit passes. + +import { EventEmitter } from "node:events"; + +const emitter = new EventEmitter(); +let ready = 0; +emitter.on("ready", () => { + ready++; + console.log("ready", ready); +}); +emitter.once("closed", function () { + console.log("closed once"); +}); + +const loose: any = emitter; +console.log(loose.emit("ready")); +console.log(loose.emit("ready", 1, "two")); +console.log(loose.emit("closed", { reason: "done" })); +console.log(loose.emit("closed")); +console.log(loose.emit("missing")); +console.log(emitter.listenerCount("ready"), emitter.listenerCount("closed"), ready); diff --git a/tests/harness/corpus-costs/6941-emitter-untyped-zero-arg-listener.ts.json b/tests/harness/corpus-costs/6941-emitter-untyped-zero-arg-listener.ts.json new file mode 100644 index 000000000..21b784250 --- /dev/null +++ b/tests/harness/corpus-costs/6941-emitter-untyped-zero-arg-listener.ts.json @@ -0,0 +1 @@ +{ "plain": 2800, "san": 3400 } From 04435a70ac2b4c6b3b60336b86fffef39e86587d Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 11:44:13 -0500 Subject: [PATCH 35/51] Address the second round of review comments on prototype toString conversions --- packages/compiler/src/backend/llvm/dyn.ts | 36 ++++++++++--------- packages/runtime/src/scr_dyn_invoke.c | 9 +++-- packages/runtime/src/scr_json.c | 4 +++ .../corpus/prototype-tostring-conversions.cjs | 12 +++++++ 4 files changed, 43 insertions(+), 18 deletions(-) diff --git a/packages/compiler/src/backend/llvm/dyn.ts b/packages/compiler/src/backend/llvm/dyn.ts index ff975a800..6911ef98b 100644 --- a/packages/compiler/src/backend/llvm/dyn.ts +++ b/packages/compiler/src/backend/llvm/dyn.ts @@ -3362,18 +3362,22 @@ export class LlDyn { B.line(`call void @scr_jb_putc(ptr %b, i8 44)`); B.br(lel); B.startBlock(lel); - // An element's toString can append to, shrink or reallocate the - // array: each index reads the current items, an index past the - // current length prints empty, and the element stays retained - // while it converts. - const now = this.lenOf(B, "%d"); - const inside = B.tmp(); - B.line(`${inside} = icmp ult ${this.S} ${i}, ${now}`); - const lIn = B.newLabel("ds.in"); - const lSkip = B.newLabel("ds.sk"); - B.condBr(inside, lIn, lSkip); - B.startBlock(lIn); - const e = this.itemAt(B, this.itemsOf(B, "%d"), i); + // Each index reads the element as join does (scr_dyn_arr_at, +1): + // from the current storage, undefined past the current length, + // and through the prototype for a hole. An element's toString + // can append to, shrink or reallocate the array; a getter that + // throws answers NULL with the exception pending. + host.declare(`declare ptr @scr_dyn_arr_at(ptr, double)`); + host.declare(`declare void @scr_dyn_release_v(ptr)`); + const index = B.tmp(); + B.line(`${index} = uitofp ${this.S} ${i} to double`); + const e = B.tmp(); + B.line(`${e} = call ptr @scr_dyn_arr_at(ptr %d, double ${index})`); + const missing = B.tmp(); + B.line(`${missing} = icmp eq ptr ${e}, null`); + const lGot = B.newLabel("ds.gt"); + B.condBr(missing, done, lGot); + B.startBlock(lGot); const ek = this.kindOf(B, e); const isU = B.tmp(); const isN = B.tmp(); @@ -3382,14 +3386,14 @@ export class LlDyn { B.line(`${isN} = icmp eq i32 ${ek}, ${DYN_KIND.NULL}`); B.line(`${unit} = or i1 ${isU}, ${isN}`); const lRec = B.newLabel("ds.rc"); + const lSkip = B.newLabel("ds.sk"); B.condBr(unit, lSkip, lRec); B.startBlock(lSkip); + B.line(`call void @scr_dyn_release_v(ptr ${e})`); brNext(); B.startBlock(lRec); - const held = this.retainDyn(B, e); - host.declare(`declare void @scr_dyn_release_v(ptr)`); - B.line(`call void @sc_ds_buf(ptr %b, ptr ${held})`); - B.line(`call void @scr_dyn_release_v(ptr ${held})`); + B.line(`call void @sc_ds_buf(ptr %b, ptr ${e})`); + B.line(`call void @scr_dyn_release_v(ptr ${e})`); // The first exception an element's conversion throws ends the // conversion; later elements are not read. const thrown = B.tmp(); diff --git a/packages/runtime/src/scr_dyn_invoke.c b/packages/runtime/src/scr_dyn_invoke.c index e413acfe9..ffe1d0330 100644 --- a/packages/runtime/src/scr_dyn_invoke.c +++ b/packages/runtime/src/scr_dyn_invoke.c @@ -1215,7 +1215,7 @@ static ScrDyn *scr_dyn_invoke_impl( scr_jb_init(&b); ScrStr *separator = argc > 0 && args[0]->kind != SCR_DYN_UNDEF ? scr_dyn_string_coerce_js(args[0]) : scr_str_new(",", 1); - if (!separator) { free(b.data); return NULL; } + if (!separator) return NULL; /* nothing written yet */ for (size_t i = 0; i < len; i++) { if (i > 0) scr_jb_write(&b, separator->data, separator->len); ScrDyn *e = scr_dyn_arr_at(recv, (double)i); @@ -1223,7 +1223,12 @@ static ScrDyn *scr_dyn_invoke_impl( if (e->kind != SCR_DYN_UNDEF && e->kind != SCR_DYN_NULL) scr_dyn_display_buf(&b, e, true); scr_dyn_release(e); } - if (!e || scr_exc_pending()) { scr_str_release(separator); free(b.data); return NULL; } + if (!e || scr_exc_pending()) { + /* The buffer's bytes live in a string allocation: discard it as one. */ + scr_str_release(scr_jb_finish(&b)); + scr_str_release(separator); + return NULL; + } } scr_str_release(separator); ScrStr *joined = scr_jb_finish(&b); diff --git a/packages/runtime/src/scr_json.c b/packages/runtime/src/scr_json.c index 13fc73187..a9acb8f53 100644 --- a/packages/runtime/src/scr_json.c +++ b/packages/runtime/src/scr_json.c @@ -7507,6 +7507,10 @@ ScrStr *scr_dyn_to_string(const ScrDyn *d, const ScrStr *enc) { out = joined; } ScrDyn *e = scr_dyn_arr_at(d, (double)i); + if (!e) { /* an inherited getter threw: the exception is pending */ + scr_str_release(out); + return NULL; + } if (e->kind == SCR_DYN_UNDEF || e->kind == SCR_DYN_NULL) { /* JS join: empty */ scr_dyn_release(e); if (scr_exc_pending()) { diff --git a/tests/corpus/prototype-tostring-conversions.cjs b/tests/corpus/prototype-tostring-conversions.cjs index a5c6ba83c..562f1ae9a 100644 --- a/tests/corpus/prototype-tostring-conversions.cjs +++ b/tests/corpus/prototype-tostring-conversions.cjs @@ -64,3 +64,15 @@ try { console.log(`${stops}`); } catch (e) { console.log("template", e.message, try { console.log(stops.join()); } catch (e) { console.log("join", e.message, visited.length); } try { console.log(String(stops)); } catch (e) { console.log("String", e.message, visited.length); } try { console.log(`${[stops]}`); } catch (e) { console.log("nested", e.message, visited.length); } + +// A hole reads through the prototype: a getter there that throws ends each +// conversion with its exception. +const holeyProto = Object.create(Array.prototype); +Object.defineProperty(holeyProto, '1', { get() { throw new Error('getter'); } }); +const holey = []; +holey[0] = 'a'; +holey[2] = 'c'; +Object.setPrototypeOf(holey, holeyProto); +try { console.log(String(holey)); } catch (e) { console.log('String', e.message); } +try { console.log(holey.join('|')); } catch (e) { console.log('join', e.message); } +try { console.log(`${holey}`); } catch (e) { console.log('template', e.message); } From cf1e79b5f9cbb3175a2924c3367550a9729ef961 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 11:49:05 -0500 Subject: [PATCH 36/51] Keep the write-target AST getters out of the generated region --- .../compiler/src/frontend/ts7/ast-node.ts | 42 +++++++++++-------- scripts/generate-ts7-ast-schema.mjs | 16 +++++-- 2 files changed, 36 insertions(+), 22 deletions(-) diff --git a/packages/compiler/src/frontend/ts7/ast-node.ts b/packages/compiler/src/frontend/ts7/ast-node.ts index 5fef740ae..f337bd623 100644 --- a/packages/compiler/src/frontend/ts7/ast-node.ts +++ b/packages/compiler/src/frontend/ts7/ast-node.ts @@ -572,6 +572,30 @@ export class AstNode { return token?.kind === SyntaxKind.QuestionToken ? token : undefined; } + // A write target keeps its parentheses and non-null assertions in the + // tree; with AstFileOptions.writeTargets these views return the target + // without them. + get initializer(): AstNode | undefined { + const initializer = this.childNode("initializer"); + return initializer !== undefined && this.file.writeTargets && writesTarget(this, "initializer") + ? unwrapTarget(initializer) + : initializer; + } + + get left(): AstNode | undefined { + const left = this.childNode("left"); + return left !== undefined && this.file.writeTargets && writesTarget(this, "left") + ? unwrapTarget(left) + : left; + } + + get operand(): AstNode | undefined { + const operand = this.childNode("operand"); + return operand !== undefined && this.file.writeTargets && writesTarget(this, "operand") + ? unwrapTarget(operand) + : operand; + } + // BEGIN GENERATED CHILD GETTERS // Generated from typescript@7.0.2; run scripts/generate-ts7-ast-schema.mjs. get argument(): AstNode | undefined { @@ -700,24 +724,12 @@ export class AstNode { get indexType(): AstNode | undefined { return this.childNode("indexType"); } - get initializer(): AstNode | undefined { - const initializer = this.childNode("initializer"); - return initializer !== undefined && this.file.writeTargets && writesTarget(this, "initializer") - ? unwrapTarget(initializer) - : initializer; - } get jsdocPropertyTags(): AstNode | undefined { return this.childNode("jsdocPropertyTags"); } get label(): AstNode | undefined { return this.childNode("label"); } - get left(): AstNode | undefined { - const left = this.childNode("left"); - return left !== undefined && this.file.writeTargets && writesTarget(this, "left") - ? unwrapTarget(left) - : left; - } get literal(): AstNode | undefined { return this.childNode("literal"); } @@ -757,12 +769,6 @@ export class AstNode { get openingFragment(): AstNode | undefined { return this.childNode("openingFragment"); } - get operand(): AstNode | undefined { - const operand = this.childNode("operand"); - return operand !== undefined && this.file.writeTargets && writesTarget(this, "operand") - ? unwrapTarget(operand) - : operand; - } get operatorToken(): AstNode | undefined { return this.childNode("operatorToken"); } diff --git a/scripts/generate-ts7-ast-schema.mjs b/scripts/generate-ts7-ast-schema.mjs index 30c506869..a1bc80117 100644 --- a/scripts/generate-ts7-ast-schema.mjs +++ b/scripts/generate-ts7-ast-schema.mjs @@ -268,10 +268,18 @@ if (process.argv.includes("--check")) { // declarations, and require every wire property to have a declared getter. const declarations = readFileSync(join(packageRoot, "dist/api/node/node.generated.d.ts"), "utf8"); const names = [...new Set(Object.values(protocol.childProperties).flat())].sort(); -// These views adapt TS7's shared postfix slot and reparsed JSDoc bodies, and -// return modifiers as a modifier list. Their implementations live outside the -// generated region in AstNode. -const adaptedGetters = new Set(["body", "modifiers", "questionToken"]); +// These views adapt TS7's shared postfix slot and reparsed JSDoc bodies, +// return modifiers as a modifier list, and return write targets without their +// wrappers. Their implementations live outside the generated region in +// AstNode. +const adaptedGetters = new Set([ + "body", + "initializer", + "left", + "modifiers", + "operand", + "questionToken", +]); const getters = names .filter((name) => !adaptedGetters.has(name)) .map((name) => { From 86f968c05f947e7051d875d5c9a7ed9c98621d43 Mon Sep 17 00:00:00 2001 From: Malte Ubl Date: Wed, 21 Oct 2026 09:53:39 -0700 Subject: [PATCH 37/51] Test the key-order pass with checked conversions and IR rewrites A shape only checked conversions build keeps today's table; one an object literal also builds takes the literal's order; a literal the IR mapper rebuilds keeps its witness. Co-Authored-By: Claude Opus 5.5 --- .../lowering/key-order-witness.test.ts | 47 ++++++++++++++++++- 1 file changed, 45 insertions(+), 2 deletions(-) diff --git a/packages/compiler/src/frontend/lowering/key-order-witness.test.ts b/packages/compiler/src/frontend/lowering/key-order-witness.test.ts index 01f632cf3..de16e5d61 100644 --- a/packages/compiler/src/frontend/lowering/key-order-witness.test.ts +++ b/packages/compiler/src/frontend/lowering/key-order-witness.test.ts @@ -1,5 +1,6 @@ import { expect, test } from "vitest"; import { F64, type IrExpr, type IrFunction, type IrStmt, type IrType } from "../../ir/ir.js"; +import { observeRecordLiteralRebuilds, transformExpr } from "../../ir/traverse.js"; import { ShapeRegistry } from "../type-mapper.js"; import { applyKeyOrderWitnesses, @@ -156,7 +157,24 @@ test("a synthesized construction pins today's table", () => { expect(f.order(ab)).toEqual(["a", "b"]); }); -test("a checked conversion pins today's table", () => { +test("a checked conversion alone pins today's table", () => { + const f = fixture(); + const ab = f.record( + [ + ["a", OPT], + ["b", F64], + ], + ["a", "b"], + ); + const parsed: IrExpr = { kind: "dynCheck", value: ref("j", { kind: "dyn" }), type: ab, loc }; + // A late write to `a` would move it after `b`; values only a checked + // conversion builds keep today's table, so parsed values in that order + // need no source view. + f.run([decl("p", parsed), set(ref("p", ab), "a"), observe(ref("p", ab))]); + expect(f.order(ab)).toEqual(["a", "b"]); +}); + +test("a shape literals and a checked conversion both build takes the literals' order", () => { const f = fixture(); const ab = f.record( [ @@ -166,8 +184,33 @@ test("a checked conversion pins today's table", () => { ["a", "b"], ); const parsed: IrExpr = { kind: "dynCheck", value: ref("j", { kind: "dyn" }), type: ab, loc }; + // The literal's values can enumerate in no other order; parsed values in + // another order keep a record-source view. f.run([observe(f.literal(ab, ["b", "a"])), observe(parsed)]); - expect(f.order(ab)).toEqual(["a", "b"]); + expect(f.order(ab)).toEqual(["b", "a"]); +}); + +test("a literal an IR rewrite rebuilds keeps its witness", () => { + const f = fixture(); + const ab = f.record( + [ + ["a", F64], + ["b", F64], + ], + ["a", "b"], + ); + const identity = { expr: (e: IrExpr): IrExpr => e, stmt: (st: IrStmt): IrStmt => st }; + const original = f.literal(ab, ["b", "a"]); + observeRecordLiteralRebuilds((from, to) => f.notes.follow(from, to)); + let rebuilt: IrExpr; + try { + rebuilt = transformExpr(original, identity); + } finally { + observeRecordLiteralRebuilds(null); + } + expect(rebuilt).not.toBe(original); + f.run([observe(rebuilt)]); + expect(f.order(ab)).toEqual(["b", "a"]); }); test("a spread contributes its source's order, and an override keeps its position", () => { From d59ec3251ea70af8c6399f08f8387de2b6a0a4c9 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 12:06:08 -0500 Subject: [PATCH 38/51] Grow small map entry arrays through 1, 2, 3, 4, 6, 8, 12 and 16 slots --- packages/runtime/src/scr_map.c | 15 +++++++++++---- packages/runtime/src/scr_map.test.c | 16 ++++++++-------- 2 files changed, 19 insertions(+), 12 deletions(-) diff --git a/packages/runtime/src/scr_map.c b/packages/runtime/src/scr_map.c index ecd1a558b..5d4935ed0 100644 --- a/packages/runtime/src/scr_map.c +++ b/packages/runtime/src/scr_map.c @@ -702,11 +702,18 @@ static void scr_map_compact(ScrMap *m, bool reindex) { * range. */ #define SCR_MAP_MAX_ENTRIES ((size_t)1 << 30) -/* The first entry gets an array of one, so a one-entry map keeps no unused - * slots; the second grows it to four, and capacity doubles after that. */ +/* The entry capacity after `cap`: 1, 2, 3, 4, 6, 8, 12, 16, then doubling. + * Most maps hold one to four entries, so a small map keeps at most a third + * of its slots unused, and appends stay amortized constant. */ +static size_t scr_map_next_ecap(size_t cap) { + if (cap < 4) return cap + 1; + if (cap < 16) return cap < 8 ? cap + 2 : cap + 4; + return cap * 2; +} + static void scr_map_grow_entries(ScrMap *m) { - size_t cap = m->ecap == 0 ? 1 : m->ecap < 4 ? 4 : m->ecap; - while (cap < m->nentries + 1) cap *= 2; + size_t cap = scr_map_next_ecap(m->ecap); + while (cap < m->nentries + 1) cap = scr_map_next_ecap(cap); if (cap > SCR_MAP_MAX_ENTRIES) scr_map_oom(); ScrMapEntry *entries = scr_mem_realloc(m->entries, cap * scr_map_entry_size(m)); if (!entries) scr_map_oom(); diff --git a/packages/runtime/src/scr_map.test.c b/packages/runtime/src/scr_map.test.c index d6f1163f4..9916c7406 100644 --- a/packages/runtime/src/scr_map.test.c +++ b/packages/runtime/src/scr_map.test.c @@ -242,17 +242,17 @@ static void modes(void) { scr_map_release(set); } -/* Small maps keep compact storage: the entries array starts with one slot - * and grows to four, and maps of up to 8 entries keep no bucket table. */ +/* Small maps keep compact storage: the entries array grows through 1, 2, + * 3, 4, 6 and 8 slots, and maps of up to 8 entries keep no bucket table. */ static void small_storage(void) { ScrMap *map = scr_map_new(SCR_MAP_KEY_F64, SCR_MAP_VAL_F64, NULL); assert(!map->entries && map->ecap == 0); - scr_map_set_f64_f64(map, 100000, 1); - assert(map->ecap == 1); - scr_map_set_f64_f64(map, 200000, 2); - assert(map->ecap == 4); - for (int i = 3; i <= 8; i++) scr_map_set_f64_f64(map, i * 100000.0, i); - assert(map->ecap == 8 && !map->nbuckets && !map->ndense); + static const uint32_t caps[] = {1, 2, 3, 4, 6, 6, 8, 8}; + for (int i = 1; i <= 8; i++) { + scr_map_set_f64_f64(map, i * 100000.0, i); + assert(map->ecap == caps[i - 1]); + } + assert(!map->nbuckets && !map->ndense); double out; for (int i = 1; i <= 8; i++) assert(scr_map_get_f64_f64(map, i * 100000.0, &out) && out == i); assert(!scr_map_has_f64(map, 50000) && !scr_map_has_f64(map, 0)); From 293facfdf643d70d47c64bc526cd54c31bb64f27 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 12:20:12 -0500 Subject: [PATCH 39/51] Group record-flow references by symbol once per name --- .../lowering/runtime-optional-records.ts | 24 ++++++++++++------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/packages/compiler/src/frontend/lowering/runtime-optional-records.ts b/packages/compiler/src/frontend/lowering/runtime-optional-records.ts index 18e136d69..7f1bb046a 100644 --- a/packages/compiler/src/frontend/lowering/runtime-optional-records.ts +++ b/packages/compiler/src/frontend/lowering/runtime-optional-records.ts @@ -81,7 +81,9 @@ export class RecordFlowClasses { /** Identifiers by name; symbols resolve only for the names of slots and * functions, so the scan asks the checker about few nodes. */ private readonly names = new Map(); - private readonly references = new Map(); + /** The identifiers of each name grouped by the symbol they resolve to, + * built on the first reference query for that name. */ + private readonly references = new Map>(); /** Names written as shorthand properties (`{ p }`): the checker resolves * those to the property, so a binding of that name stays open. */ private readonly shorthandNames = new Set(); @@ -349,14 +351,20 @@ export class RecordFlowClasses { /** The identifiers that name `symbol`, its declaration included. */ private referencesOf(symbol: ts.Symbol): readonly ts.Identifier[] { - let list = this.references.get(symbol); - if (!list) { - list = (this.names.get(ts.symbolName(symbol)) ?? []).filter( - (node) => this.host.symbolOf(node) === symbol, - ); - this.references.set(symbol, list); + const name = ts.symbolName(symbol); + let bySymbol = this.references.get(name); + if (!bySymbol) { + bySymbol = new Map(); + for (const node of this.names.get(name) ?? []) { + const resolved = this.host.symbolOf(node); + if (!resolved) continue; + const list = bySymbol.get(resolved); + if (list) list.push(node); + else bySymbol.set(resolved, [node]); + } + this.references.set(name, bySymbol); } - return list; + return bySymbol.get(symbol) ?? []; } private collectFunctions(): void { From d8265b220922c8d756a230de7423f3facfa51c37 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 12:35:58 -0500 Subject: [PATCH 40/51] Link scr_bytes_dyn.c into libraries --- packages/compiler/src/backend/library-pack.ts | 1 + .../compiler/src/backend/native/contracts.ts | 3 + .../compiler/src/backend/native/library.ts | 1 + packages/compiler/src/index.ts | 2 + .../compiler/src/library/library-cache.ts | 2 + .../runtime-pack-common/library-matrix.mjs | 1 + packages/runtime/src/scr_bytes.c | 60 +++++++++++++++++++ packages/runtime/src/scr_bytes_io.c | 60 ------------------- 8 files changed, 70 insertions(+), 60 deletions(-) diff --git a/packages/compiler/src/backend/library-pack.ts b/packages/compiler/src/backend/library-pack.ts index 8f6dbb221..805198df1 100644 --- a/packages/compiler/src/backend/library-pack.ts +++ b/packages/compiler/src/backend/library-pack.ts @@ -33,6 +33,7 @@ export async function compilePackedLibrary( assert: options.assert ?? false, inspect: options.inspect ?? false, dynInvoke: options.dynInvoke ?? false, + ...(options.dynBuffer === true ? { dynBuffer: true } : {}), ...(options.dynTier === true ? { dynTier: true } : {}), dc: false, dynAsync: false, diff --git a/packages/compiler/src/backend/native/contracts.ts b/packages/compiler/src/backend/native/contracts.ts index a3c0a62fe..06a41c9b2 100644 --- a/packages/compiler/src/backend/native/contracts.ts +++ b/packages/compiler/src/backend/native/contracts.ts @@ -321,6 +321,9 @@ export interface LibArchiveOptions { threadInstances?: boolean; /** IR-detected link gates (the compileC precedent, refusal-narrowed). */ dynInvoke?: boolean; + /** Buffer.prototype methods on checked-dynamic values: scr_bytes_dyn.c, + * with dynInvoke. */ + dynBuffer?: boolean; /** See CompileOptions.dynTier: the scr_js_* units. */ dynTier?: boolean; regex?: boolean; diff --git a/packages/compiler/src/backend/native/library.ts b/packages/compiler/src/backend/native/library.ts index 3907fd5d8..f04e9926c 100644 --- a/packages/compiler/src/backend/native/library.ts +++ b/packages/compiler/src/backend/native/library.ts @@ -112,6 +112,7 @@ export async function compileLibArchive(opts: LibArchiveOptions): Promise ...(opts.assert && opts.bigint ? ["scr_bigint_assert.c"] : []), ...(opts.emitter ? ["scr_events_emitter.c", "scr_dyn_handle.c"] : []), ...(opts.dynInvoke ? ["scr_dyn_invoke.c"] : []), + ...(opts.dynInvoke && opts.dynBuffer ? ["scr_bytes_dyn.c"] : []), ...(opts.dynTier === true ? ["scr_js_value.c", "scr_js_object.c"] : []), ...(opts.zlib ? ["scr_zlib.c"] : []), ...(opts.copying ? ["scr_copying.c"] : []), diff --git a/packages/compiler/src/index.ts b/packages/compiler/src/index.ts index 461a32c90..08457f86b 100644 --- a/packages/compiler/src/index.ts +++ b/packages/compiler/src/index.ts @@ -1706,6 +1706,7 @@ function libraryNativeFeatures(mod: IrModule, backend: "llvm"): EarlyLibraryNati copying: features.copying, textDecoderLegacy: features.legacyTextDecoder, dynInvoke: features.dynInvoke, + ...(features.dynBuffer ? { dynBuffer: true } : {}), ...(features.dynTier ? { dynTier: true } : {}), ...(mod.lib?.identity !== undefined ? { buildId: mod.lib.identity.buildId } : {}), }; @@ -1800,6 +1801,7 @@ async function compileLibraryNative( copying: features.copying, textDecoderLegacy: features.textDecoderLegacy, dynInvoke: features.dynInvoke, + ...(features.dynBuffer === true ? { dynBuffer: true } : {}), ...(features.dynTier === true ? { dynTier: true } : {}), }; if (packTarget !== null) await compilePackedLibrary(archiveOptions, packTarget); diff --git a/packages/compiler/src/library/library-cache.ts b/packages/compiler/src/library/library-cache.ts index 356056dd8..3e77ccced 100644 --- a/packages/compiler/src/library/library-cache.ts +++ b/packages/compiler/src/library/library-cache.ts @@ -63,6 +63,8 @@ interface EarlyLibraryCacheStamp { export interface EarlyLibraryNativeFeatures { backend: "llvm"; dynInvoke: boolean; + /** Buffer.prototype methods on checked-dynamic values (scr_bytes_dyn.c). */ + dynBuffer?: boolean; dynTier?: boolean; regex: boolean; assert: boolean; diff --git a/packages/runtime-pack-common/library-matrix.mjs b/packages/runtime-pack-common/library-matrix.mjs index 667f9a34f..d9878ec7c 100644 --- a/packages/runtime-pack-common/library-matrix.mjs +++ b/packages/runtime-pack-common/library-matrix.mjs @@ -15,6 +15,7 @@ export function withLibraryRuntimeFlavors(matrix) { "scr_zlib.c", "scr_copying.c", "scr_dyn_invoke.c", + "scr_bytes_dyn.c", "scr_js_value.c", "scr_js_object.c", ]); diff --git a/packages/runtime/src/scr_bytes.c b/packages/runtime/src/scr_bytes.c index 69e26e5d7..7a6a76cf9 100644 --- a/packages/runtime/src/scr_bytes.c +++ b/packages/runtime/src/scr_bytes.c @@ -3957,3 +3957,63 @@ ScrStr *scr_bytes_join(const ScrBytes *b, const ScrStr *separator) { scr_arr_release(values); return out; } + +/* ── the checked-dynamic Buffer compare/equals validators ────────────── + * Node's argument ladders for buf.equals / buf.compare / Buffer.compare + * over dyn-boxed arguments (the invalid-input probes: string needles, + * '0' offsets, null/object range args). A well-typed dyn still computes + * the real answer — validation, not a constant fence. */ + +/* A bytes payload or the API's own ERR_INVALID_ARG_TYPE (borrowed). */ +static ScrBytes *scr_bytes_chk_u8(const ScrDyn *d, const char *argname) { + if (!scr_dyn_bytes_is(d, SCR_BYTES_U8)) { + scr_dyn_arg_type_fail(argname, "an instance of Buffer or Uint8Array", d); + return NULL; + } + return d->v.bytes; +} + +double scr_buffer_compare_chk(const ScrDyn *a, const ScrDyn *b) { + ScrBytes *b1 = scr_bytes_chk_u8(a, "buf1"); + if (!b1) return 0; + ScrBytes *b2 = scr_bytes_chk_u8(b, "buf2"); + if (!b2) return 0; + return scr_bytes_compare(b1, b2, 0, 0, 0, 0, 0); +} + +bool scr_bytes_equals_chk(const ScrBytes *recv, const ScrDyn *other) { + ScrBytes *o = scr_bytes_chk_u8(other, "otherBuffer"); + if (!o) return false; + return scr_bytes_equals(recv, o); +} + +/* One offset slot: undefined takes the Node default, non-numbers throw + * ERR_INVALID_ARG_TYPE "of type number", numbers run validateOffset. */ +static bool scr_bytes_chk_off(const ScrDyn *d, const char *name, double max, + double dflt, double *out) { + if (d->kind == SCR_DYN_UNDEF) { + *out = dflt; + return true; + } + if (d->kind != SCR_DYN_NUM) { + scr_dyn_arg_type_fail(name, "of type number", d); + return false; + } + *out = d->v.num; + return scr_bytes_validate_off(name, *out, max); +} + +double scr_bytes_compare_chk(const ScrBytes *src, const ScrDyn *target, + const ScrDyn *ts, const ScrDyn *te, + const ScrDyn *ss, const ScrDyn *se) { + ScrBytes *t = scr_bytes_chk_u8(target, "target"); + if (!t) return 0; + double tsv, tev, ssv, sev; + if (!scr_bytes_chk_off(ts, "targetStart", 9007199254740991.0, 0, &tsv)) return 0; + if (!scr_bytes_chk_off(te, "targetEnd", (double)t->len, (double)t->len, &tev)) return 0; + if (!scr_bytes_chk_off(ss, "sourceStart", 9007199254740991.0, 0, &ssv)) return 0; + if (!scr_bytes_chk_off(se, "sourceEnd", (double)src->len, (double)src->len, &sev)) return 0; + /* Every slot validated or defaulted above; nargs 4 revalidates the + * now-known-good numbers (a no-op) and keeps one comparison core. */ + return scr_bytes_compare(src, t, 4, tsv, tev, ssv, sev); +} diff --git a/packages/runtime/src/scr_bytes_io.c b/packages/runtime/src/scr_bytes_io.c index c852388e9..eb3661fc6 100644 --- a/packages/runtime/src/scr_bytes_io.c +++ b/packages/runtime/src/scr_bytes_io.c @@ -1368,66 +1368,6 @@ ScrBytes *scr_buffer_from_dyn_ref_pooled(const ScrDyn *value, const ScrStr *enco return scr_buffer_from_dyn_pooled(value, encoding); } -/* ── the checked-dynamic Buffer compare/equals validators ────────────── - * Node's argument ladders for buf.equals / buf.compare / Buffer.compare - * over dyn-boxed arguments (the invalid-input probes: string needles, - * '0' offsets, null/object range args). A well-typed dyn still computes - * the real answer — validation, not a constant fence. */ - -/* A bytes payload or the API's own ERR_INVALID_ARG_TYPE (borrowed). */ -static ScrBytes *scr_bytes_chk_u8(const ScrDyn *d, const char *argname) { - if (!scr_dyn_bytes_is(d, SCR_BYTES_U8)) { - scr_dyn_arg_type_fail(argname, "an instance of Buffer or Uint8Array", d); - return NULL; - } - return d->v.bytes; -} - -double scr_buffer_compare_chk(const ScrDyn *a, const ScrDyn *b) { - ScrBytes *b1 = scr_bytes_chk_u8(a, "buf1"); - if (!b1) return 0; - ScrBytes *b2 = scr_bytes_chk_u8(b, "buf2"); - if (!b2) return 0; - return scr_bytes_compare(b1, b2, 0, 0, 0, 0, 0); -} - -bool scr_bytes_equals_chk(const ScrBytes *recv, const ScrDyn *other) { - ScrBytes *o = scr_bytes_chk_u8(other, "otherBuffer"); - if (!o) return false; - return scr_bytes_equals(recv, o); -} - -/* One offset slot: undefined takes the Node default, non-numbers throw - * ERR_INVALID_ARG_TYPE "of type number", numbers run validateOffset. */ -static bool scr_bytes_chk_off(const ScrDyn *d, const char *name, double max, - double dflt, double *out) { - if (d->kind == SCR_DYN_UNDEF) { - *out = dflt; - return true; - } - if (d->kind != SCR_DYN_NUM) { - scr_dyn_arg_type_fail(name, "of type number", d); - return false; - } - *out = d->v.num; - return scr_bytes_validate_off(name, *out, max); -} - -double scr_bytes_compare_chk(const ScrBytes *src, const ScrDyn *target, - const ScrDyn *ts, const ScrDyn *te, - const ScrDyn *ss, const ScrDyn *se) { - ScrBytes *t = scr_bytes_chk_u8(target, "target"); - if (!t) return 0; - double tsv, tev, ssv, sev; - if (!scr_bytes_chk_off(ts, "targetStart", 9007199254740991.0, 0, &tsv)) return 0; - if (!scr_bytes_chk_off(te, "targetEnd", (double)t->len, (double)t->len, &tev)) return 0; - if (!scr_bytes_chk_off(ss, "sourceStart", 9007199254740991.0, 0, &ssv)) return 0; - if (!scr_bytes_chk_off(se, "sourceEnd", (double)src->len, (double)src->len, &sev)) return 0; - /* Every slot validated or defaulted above; nargs 4 revalidates the - * now-known-good numbers (a no-op) and keeps one comparison core. */ - return scr_bytes_compare(src, t, 4, tsv, tev, ssv, sev); -} - /* new Buffer(number, encoding) — the deprecated ctor's string arm with a * non-string first argument: Node's exact ERR_INVALID_ARG_TYPE (the * throwing path never fires DEP0005, so compiled silence matches). */ From 1bf49286d1aba8c6cb505d2704a2519a210a556a Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 12:50:36 -0500 Subject: [PATCH 41/51] Document when builds use multiple cores How It Works gains a Build parallelism section: the frontend runs on one thread, executables divide into partitions by the size of their LLVM text, the LLVM helper compiles partitions on a thread per hardware thread, what dev and optimized builds run in parallel and in order, what a rebuild compiles again, and what profile-guided optimization adds to build time. The CLI reference links to it from --emit, --optimization, --profile-use, SCRIPTC_CACHE_DIR, and Profile-guided optimization. Fixes #1340 Co-authored-by: Quino Terrasa <26749475+espetro@users.noreply.github.com> --- docs/content/docs/cli.mdx | 10 ++--- docs/content/docs/how-it-works.mdx | 61 +++++++++++++++++++++++++++++- 2 files changed, 65 insertions(+), 6 deletions(-) diff --git a/docs/content/docs/cli.mdx b/docs/content/docs/cli.mdx index 9b6e88628..daeb37ea0 100644 --- a/docs/content/docs/cli.mdx +++ b/docs/content/docs/cli.mdx @@ -109,7 +109,7 @@ With `--sparse`, the profile leaves out the functions the runs never reached. Th
Primary artifact path. An explicit path is exact. Defaults are .scriptc/<name>.ir.json, .ll, .s, .o, or the platform executable name.
--emit <ir|llvm|asm|obj|exe>
-
Select the primary artifact: serialized typed IR (ir), textual LLVM IR (llvm), assembly (asm), a relocatable object (obj), or an executable (exe, the default). Every kind except exe is produced without an external compiler, linker, or SDK.
+
Select the primary artifact: serialized typed IR (ir), textual LLVM IR (llvm), assembly (asm), a relocatable object (obj), or an executable (exe, the default). Every kind except exe is produced without an external compiler, linker, or SDK. Assembly and object output is compiled as one module; only executables are divided into partitions that compile in parallel (see Build parallelism).
--print <native-link-info|diagnostics>
native-link-info: build an object (equivalent to --emit=obj) and print its machine-readable external link recipe as JSON instead of printing the artifact path. The document names the exact installed precompiled runtime pack and all link inputs, but does not invoke a linker. diagnostics: for build and coverage, print the versioned JSON diagnostics document described in Coverage Reports instead of the human output.
@@ -127,7 +127,7 @@ With `--sparse`, the profile leaves out the functions the runs never reached. Th
Remove symbol and debug payload when linking an executable, including cross-compiled targets. This flag preserves code optimization.
--optimization <release|dev|speed>
-
Select native optimization for executable, assembly, or object output. The default is release (-O2), which emits string retains and the retains and releases of dynamic values inline and lets LLVM inline small runtime functions that keep no internal runtime state into executable code; programs large enough to be compiled in parallel partitions skip that inlining to keep build time and size in check. dev uses -O0 and emits source locations for TypeScript and JavaScript breakpoints and stack frames. LLVM builds include source variable names and native storage descriptions; see Native debugging for the supported representations. macOS executable builds also produce an adjacent .dSYM bundle. speed is -O2 plus optimizations that trade binary size and build time for run-time speed: it lets LLVM inline more small runtime functions into program code, emits the remaining reference counting inline, and on x86-64 starts every runtime function on a cache-line boundary. Expect larger executables and longer cold builds than release; how much faster a program runs depends on how much of its time is spent in the runtime. Runtime inlining needs a runtime pack built with the LLVM helper; where a runtime pack lacks it, programs link the same runtime without it and speed links the release runtime and keeps inline reference counting. Selecting speed never changes what release builds produce, and the two never share cached artifacts. Library profiles accept "optimization": "speed" for inline reference counting.
+
Select native optimization for executable, assembly, or object output. The default is release (-O2), which emits string retains and the retains and releases of dynamic values inline and lets LLVM inline small runtime functions that keep no internal runtime state into executable code; programs large enough to be compiled in parallel partitions skip that inlining to keep build time and size in check. dev uses -O0 and emits source locations for TypeScript and JavaScript breakpoints and stack frames. LLVM builds include source variable names and native storage descriptions; see Native debugging for the supported representations. macOS executable builds also produce an adjacent .dSYM bundle. speed is -O2 plus optimizations that trade binary size and build time for run-time speed: it lets LLVM inline more small runtime functions into program code, emits the remaining reference counting inline, and on x86-64 starts every runtime function on a cache-line boundary. Expect larger executables and longer cold builds than release; how much faster a program runs depends on how much of its time is spent in the runtime. Runtime inlining needs a runtime pack built with the LLVM helper; where a runtime pack lacks it, programs link the same runtime without it and speed links the release runtime and keeps inline reference counting. Selecting speed never changes what release builds produce, and the two never share cached artifacts. Library profiles accept "optimization": "speed" for inline reference counting. In executables large enough to be divided into partitions, dev compiles each partition on its own, and release and speed add a step that lets partitions import small functions from each other; see Build parallelism.
--windows-subsystem <console|gui>
Select the subsystem of a Windows executable. console is the default; gui prevents Windows from opening a console window for the app. Valid only for executable builds targeting Windows.
@@ -139,7 +139,7 @@ With `--sparse`, the profile leaves out the functions the runs never reached. Th
On Linux, build an instrumented executable for profile-guided optimization. Each run writes a raw profile to the path pattern, where %p expands to the process ID; the LLVM_PROFILE_FILE environment variable overrides the pattern at run time. A relative pattern resolves from the current working directory. Instrumented programs run slower. Valid only for executable builds with --optimization=release or speed.
--profile-use <file.profdata>
-
Optimize the executable with a profile from scriptc merge-profiles: code is laid out, inlined, and split by how the profiled runs used it. Code the profile does not describe, or describes from an older version of the program, is optimized as without a profile, so a stale profile only loses part of its gain. Valid only for executable builds with --optimization=release or speed.
+
Optimize the executable with a profile from scriptc merge-profiles: code is laid out, inlined, and split by how the profiled runs used it. Code the profile does not describe, or describes from an older version of the program, is optimized as without a profile, so a stale profile only loses part of its gain. A new or changed profile compiles every partition of the program again. Valid only for executable builds with --optimization=release or speed.
--npm-static <pkg[,pkg…]|auto>
Name npm packages to compile even if they don't ship their own type declarations (repeatable). A trailing ..., as in fs-extra..., also names every package that package depends on. auto uses the default automatic selection. Packages that can't compile produce diagnostics. See npm Dependencies.
@@ -182,7 +182,7 @@ With `--sparse`, the profile leaves out the functions the runs never reached. Th
SCRIPTC_CACHE_DIR
-
Override the persistent build-cache root. By default it is $XDG_CACHE_HOME/scriptc/build when set, $HOME/Library/Caches/scriptc/build on macOS, %LOCALAPPDATA%\scriptc\cache\build on Windows, and $HOME/.cache/scriptc/build elsewhere. Cached frontend results and LLVM artifacts are verified against their source, helper, target, and compilation settings. Native outputs also depend on the selected runtime pack and linker inputs. FFI objects, archives, and system libraries relink against their current dependencies. Sanitizer development builds additionally cache instrumented runtime objects and verify external compiler and header inputs. An existing POSIX override must already be private; otherwise caching is bypassed without changing the directory's permissions.
+
Override the persistent build-cache root. By default it is $XDG_CACHE_HOME/scriptc/build when set, $HOME/Library/Caches/scriptc/build on macOS, %LOCALAPPDATA%\scriptc\cache\build on Windows, and $HOME/.cache/scriptc/build elsewhere. Cached frontend results and LLVM artifacts, including the compiled partitions of large executables, are verified against their source, helper, target, and compilation settings. Native outputs also depend on the selected runtime pack and linker inputs. FFI objects, archives, and system libraries relink against their current dependencies. Sanitizer development builds additionally cache instrumented runtime objects and verify external compiler and header inputs. An existing POSIX override must already be private; otherwise caching is bypassed without changing the directory's permissions.
SCRIPTC_NO_CACHE
Set to 1 to bypass all build-cache reads and writes. An explicitly empty SCRIPTC_CACHE_DIR has the same effect.
@@ -223,7 +223,7 @@ $ scriptc merge-profiles -o app.profdata profiles/*.profraw $ scriptc build app.ts --optimization=speed --profile-use app.profdata ``` -Train on workloads like the ones the program will run: code the training never reached is optimized as without a profile. The profile can be committed and reused as the program changes. Functions whose control flow changed since the profile was collected lose their profile, and the rest keep theirs, so regenerate it when much of the program has changed. +Train on workloads like the ones the program will run: code the training never reached is optimized as without a profile. The profile can be committed and reused as the program changes. Functions whose control flow changed since the profile was collected lose their profile, and the rest keep theirs, so regenerate it when much of the program has changed. [Build parallelism](/docs/how-it-works#build-parallelism) describes what a profile adds to build time. ## LLVM backend diff --git a/docs/content/docs/how-it-works.mdx b/docs/content/docs/how-it-works.mdx index 5c6d7be63..f1b64948d 100644 --- a/docs/content/docs/how-it-works.mdx +++ b/docs/content/docs/how-it-works.mdx @@ -27,11 +27,70 @@ Runtime pack and linker -> executable 1. **Frontend:** TypeScript parses and checks the program against `es2025`, scriptc's ambient declarations, and available project declarations. `tsconfig.json` controls checker strictness. The compiler uses the checked types and narrowing information to lower supported operations; unsupported operations produce diagnostics. JavaScript files that TypeScript checks (a `// @ts-check` comment, or `checkJs` in `tsconfig.json`) report values that do not match their JSDoc types as errors, and their classes use those types the way TypeScript classes use annotations. 2. **Typed IR:** A validated, serializable intermediate representation connects the frontend and backend. Types are concrete: generic functions are specialized, unions use tags, and closures have explicit captures. `--emit=ir` writes this representation as JSON. -3. **LLVM backend:** `--emit=llvm` writes textual LLVM IR. The matching LLVM 22 helper generates assembly or objects on supported macOS, Linux, and Windows hosts, including output targeting WASI. Executable and library builds use this backend. Large executables, native and WASI alike, are divided into partitions that are optimized and compiled in parallel; optimized builds summarize every partition first so small functions can still be inlined across partitions. The partition count depends only on the program, so output is the same on every host, and a rebuild reuses the compiled partitions that an edit does not affect. +3. **LLVM backend:** `--emit=llvm` writes textual LLVM IR. The matching LLVM 22 helper generates assembly or objects on supported macOS, Linux, and Windows hosts, including output targeting WASI. Executable and library builds use this backend. Large executables, native and WASI alike, are divided into partitions that are optimized and compiled in parallel; see [Build parallelism](#build-parallelism). 4. **Linking:** Runtime packs contain the precompiled C runtime and vendored libraries. A hashed manifest selects the artifacts the program needs for its features, optimization mode, and (for libraries) state model. The target linker and SDK or sysroot produce the executable. Sanitizer builds use an instrumented C runtime. `comptime` calls run during the frontend stage in an isolated evaluator inside the compiler, based on QuickJS. Their results are embedded in the program as constants. +## Build parallelism + +Most of a build runs on one CPU core. The LLVM helper uses more cores only when it compiles an executable in partitions. + +### Frontend + +Parsing, type checking, lowering, and LLVM IR generation run one after another on one thread. The type checker runs in a separate process that the compiler waits on. When none of a build's inputs has changed since an earlier build, the frontend's results come from the build cache and this stage is skipped. A change to any input runs the whole stage again. + +### Partitions + +The LLVM helper divides an executable's LLVM IR into partitions by its size: one partition per 4 MiB of LLVM text, rounded up to a power of two, and at most 64. `--emit=llvm` writes this text, so its file size shows how many partitions a program gets. + + + + + + + + + + + + + + +
LLVM textPartitions
Up to 4 MiB1
Over 4 MiB, up to 8 MiB2
Over 8 MiB, up to 16 MiB4
Over 16 MiB, up to 32 MiB8
Over 32 MiB, up to 64 MiB16
Over 64 MiB, up to 128 MiB32
Over 128 MiB64
+ +The count depends only on the size of the LLVM text, so a program has the same partitions, and builds the same executable, on every host. A function's partition follows from its name, and a small internal function goes with the one function that calls it, so an edit leaves the other functions in their partitions. + +These builds are not divided into partitions and compile on one thread: `--emit=asm`, `--emit=obj`, and `--print=native-link-info` output, library builds (`--lib`; `dev` library builds are compiled in pieces, one after another), and `--sanitize` builds, which compile the LLVM IR with the C compiler. + +The helper compiles partitions on a pool with one thread per hardware thread of the host, so an executable with two partitions uses at most two cores in the backend. A program of one partition is optimized and compiled on one thread. + +How partitions are compiled depends on `--optimization`: + +- `dev` compiles each partition on its own, all in parallel. +- `release` and `speed` optimize each partition and write a summary of its functions, all in parallel. Then a serial step reads every summary and decides which small functions each partition imports from the others, so that calls between partitions can still be inlined. Last, the partitions are optimized with their imports and compiled, again in parallel. + +Reading the LLVM text and dividing it into partitions, and the import step, run on one thread. + +### Rebuilds + +With two or more partitions, the helper keeps each partition's summary and object in the build cache (see `SCRIPTC_CACHE_DIR` in the [CLI reference](/docs/cli#environment-variables)). A rebuild after an edit compiles again only the partitions whose functions changed and, in `release` and `speed` builds, the partitions that import a changed function; the others come from the cache. Dividing the program, the import step, and linking run on every build. A cold build, with nothing cached, compiles every partition. + +A program of one partition has no partition cache: a rebuild after an edit compiles the whole program again, and a rebuild that leaves the LLVM text unchanged takes the compiled code from the build cache. + +When the LLVM text grows past a partition boundary (4, 8, 16 MiB, and so on), the partition count doubles, functions move to new partitions, and the next build compiles every partition again. + +### Profile-guided optimization + +Building with a profile adds these costs (see [Profile-guided optimization](/docs/cli#profile-guided-optimization)): + +- `--profile-generate` builds an instrumented executable, which runs slower while it records the profile. +- The profile's contents are part of every partition's cache key. A new or changed profile compiles every partition again; builds without a profile keep their own cache entries. +- With a profile, the import step lets a hot call import a function of up to twice the usual size limit. LLVM's default limits for profiled calls are 10 and 100 times as large, and on large programs they tripled the build's time and memory without making the program measurably faster. +- The profile does not change which partition a function is in. + +Collecting a profile takes an instrumented build, the training runs, and the merge. With a committed profile, later builds skip those steps and are one optimized build each. + ## Inspect compiler output The following examples use this program: From 96f31cbc1838044295653308be2392fce6cca757 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 13:04:07 -0500 Subject: [PATCH 42/51] Shrink the array header to 32 bytes An array's element ops are a registry id in `spare` instead of a pointer field: descriptors get their id on their first array (emitted descriptors are writable for it), and a program with more element types than the id holds keeps the rest in the array's out-of-line storage. The out-of-line storage (sparse indices, noncanonical numeric properties, match metadata) is no longer a header field. An array that has it keeps its pointer in the 8 bytes before its dense allocation and sets SCR_ARR_HAS_EXT; such an array always has a dense allocation of its own (inline slots move out, an empty array gets the prefix alone), and every allocation, reallocation and free of the dense storage accounts for the prefix. Loops over elements look the ops up once, and the state lookup below the dense capacity reads no out-of-line storage. --- .../compiler/src/backend/llvm/array-search.ts | 3 +- packages/compiler/src/backend/llvm/common.ts | 4 +- .../compiler/src/backend/llvm/emit-module.ts | 18 +- .../src/backend/llvm/expr-stream-bridges.ts | 5 +- packages/compiler/src/backend/llvm/shapes.ts | 7 +- packages/runtime/src/scr_array.c | 350 ++++++++++++------ packages/runtime/src/scr_array.h | 91 ++++- packages/runtime/src/scr_async_dyn.c | 2 +- packages/runtime/src/scr_bytes_io.c | 4 +- packages/runtime/src/scr_copying.c | 4 +- packages/runtime/src/scr_cycle.c | 2 +- packages/runtime/src/scr_events_emitter.c | 2 +- packages/runtime/src/scr_itable.c | 10 +- packages/runtime/src/scr_json.c | 2 +- packages/runtime/src/scr_json.test.c | 2 +- packages/runtime/src/scr_lib.c | 2 +- packages/runtime/src/scr_map.c | 2 +- packages/runtime/src/scr_publish.c | 8 +- packages/runtime/src/scr_regex.c | 10 +- packages/runtime/src/scr_region.c | 14 +- packages/runtime/src/scr_region.test.c | 2 +- packages/runtime/src/scr_stream.c | 2 +- packages/runtime/test/test_array.c | 66 +++- packages/runtime/test/test_lib.c | 2 +- packages/runtime/test/test_map.c | 2 +- packages/runtime/test/test_regex.c | 2 +- 26 files changed, 441 insertions(+), 177 deletions(-) diff --git a/packages/compiler/src/backend/llvm/array-search.ts b/packages/compiler/src/backend/llvm/array-search.ts index 74d79f7be..d61d91e5c 100644 --- a/packages/compiler/src/backend/llvm/array-search.ts +++ b/packages/compiler/src/backend/llvm/array-search.ts @@ -23,6 +23,7 @@ * is little-endian, which the short-string words assume. */ import type { IrType } from "../../ir/ir.js"; +import { SCR_ARR_FIELD } from "./common.js"; import { FN_ATTRS } from "./shapes.js"; export type ArraySearchKind = "ptr" | "str"; @@ -72,7 +73,7 @@ export function emitArraySearchHelpers( ` %dense = icmp eq ${sz} %f, %len`, ` br i1 %dense, label %scan, label %slow`, `scan:`, - ` %dp = getelementptr inbounds %ScrArr, ptr %a, i32 0, i32 8`, + ` %dp = getelementptr inbounds %ScrArr, ptr %a, i32 0, i32 ${SCR_ARR_FIELD.data}`, ` %data = load ptr, ptr %dp`, ); if (kind === "ptr") { diff --git a/packages/compiler/src/backend/llvm/common.ts b/packages/compiler/src/backend/llvm/common.ts index e6fc56f6f..822ee29e2 100644 --- a/packages/compiler/src/backend/llvm/common.ts +++ b/packages/compiler/src/backend/llvm/common.ts @@ -21,9 +21,7 @@ export const SCR_ARR_FIELD = { cyc: 4, spare: 5, filled: 6, - ops: 7, - data: 8, - ext: 9, + data: 7, } as const; /** Loads the i32 `len` or `cap` of a %ScrArr at `ptr` into `dest` as a diff --git a/packages/compiler/src/backend/llvm/emit-module.ts b/packages/compiler/src/backend/llvm/emit-module.ts index 732a21ee6..17b27fe8d 100644 --- a/packages/compiler/src/backend/llvm/emit-module.ts +++ b/packages/compiler/src/backend/llvm/emit-module.ts @@ -439,15 +439,17 @@ export function emitModuleParts(emitter: LlEmitter): string[] { `%ScrFfiTable = type { ptr, ${emitter.sizeType}, ${emitter.sizeType}, ptr, i8, ptr, ptr, ${emitter.sizeType}, ${emitter.sizeType}, ${emitter.sizeType}, ptr, ptr }`, `%ScrRegex = type { ${emitter.sizeType}, ptr, ptr, ptr, double, ptr, ptr, ptr }`, // ScrArr mirrors scr_runtime.h field-for-field: { rc, len (i32), cap - // (i32), elem, cyc, spare, filled, ops, data, ext } (field numbers in + // (i32), elem, cyc, spare, filled, data } (field numbers in // SCR_ARR_FIELD), where the dense states follow the cap value slots at - // data, filled bounds the hole-free prefix, ops points at the element type's constant RC descriptor - // (%ScrArrOps, SCR_ELEM_REF only) and ext at the out-of-line sparse, - // property and metadata storage (NULL until first used). Live dynamic - // stream commits swap its dense, presence and out-of-line storage while + // data, filled bounds the hole-free prefix, and spare holds the storage + // flags and the registry id of the element type's RC descriptor + // (%ScrArrOps, SCR_ELEM_REF only; the runtime assigns ids). An array + // with out-of-line sparse, property or metadata storage keeps a pointer + // to it in the word before its dense allocation. Live dynamic stream + // commits swap its dense, presence and out-of-line storage while // preserving the target object's identity. - `%ScrArr = type { ${emitter.sizeType}, i32, i32, i8, i8, i16, i32, ptr, ptr, ptr }`, - `%ScrArrOps = type { ptr, ptr, ptr }`, + `%ScrArr = type { ${emitter.sizeType}, i32, i32, i8, i8, i16, i32, ptr }`, + `%ScrArrOps = type { ptr, ptr, ptr, i32 }`, // The ScrMap prefix { rc, key_kind, val_kind, traced } — the inline // RC fast paths test the traced byte (shapes.ts). ...(emitter.rcHelpers === null ? [] : [`%ScrMapRc = type { ${emitter.sizeType}, i8, i8, i8 }`]), @@ -709,7 +711,7 @@ export function emitModuleParts(emitter: LlEmitter): string[] { wide ? `ptr ${s}` : `<{ ptr, i32 }> <{ ptr ${s}, i32 0 }>`; out.push( `@${inst.sym}_data = internal constant <{ [${n} x ${slot}], [${n} x i8] }> <{ [${n} x ${slot}] [ ${inst.slots.map(slotValue).join(", ")} ], [${n} x i8] ${present} }>, align 8`, - `@${inst.sym} = internal global %ScrArr { ${emitter.sizeType} -1, i32 ${n}, i32 ${n}, i8 2, i8 0, i16 0, i32 ${n}, ptr null, ptr @${inst.sym}_data, ptr null }`, + `@${inst.sym} = internal global %ScrArr { ${emitter.sizeType} -1, i32 ${n}, i32 ${n}, i8 2, i8 0, i16 0, i32 ${n}, ptr @${inst.sym}_data }`, ); } if (emitter.templateStringsInstances.size > 0) out.push(``); diff --git a/packages/compiler/src/backend/llvm/expr-stream-bridges.ts b/packages/compiler/src/backend/llvm/expr-stream-bridges.ts index 6b51f9afa..ce33aa2cd 100644 --- a/packages/compiler/src/backend/llvm/expr-stream-bridges.ts +++ b/packages/compiler/src/backend/llvm/expr-stream-bridges.ts @@ -193,7 +193,10 @@ export function streamTypedRefCommitAdapter( { name: "cap", index: SCR_ARR_FIELD.cap, type: "i32" }, { name: "data", index: SCR_ARR_FIELD.data, type: "ptr" }, { name: "filled", index: SCR_ARR_FIELD.filled, type: "i32" }, - { name: "ext", index: SCR_ARR_FIELD.ext, type: "ptr" }, + // The out-of-line storage's pointer lives before the dense allocation + // and moves with data; its flag is in spare, whose element ops id the + // two arrays share. + { name: "spare", index: SCR_ARR_FIELD.spare, type: "i16" }, ]; for (const member of storageMembers) { lines.push( diff --git a/packages/compiler/src/backend/llvm/shapes.ts b/packages/compiler/src/backend/llvm/shapes.ts index 8844921c5..58295e42b 100644 --- a/packages/compiler/src/backend/llvm/shapes.ts +++ b/packages/compiler/src/backend/llvm/shapes.ts @@ -1074,15 +1074,16 @@ export function arrNewCall( return `call ptr @scr_arr_new_ops(ptr ${arrOps(host, elem)}, ${host.sizeType} ${capText})`; } -/** The constant %ScrArrOps descriptor (retain, release, trace) of a REF - * element type, defined once per module and shared by all its arrays. */ +/** The %ScrArrOps descriptor (retain, release, trace, registry id) of a REF + * element type, defined once per module and shared by all its arrays. It is + * writable: the runtime stores its registry id on the first array. */ export function arrOps(host: ShapeHost, elem: IrType): string { const v = vAdapters(host, elem); const trace = traceArg(host, elem); const part = (sym: string): string => sym.replace(/^@/, "").replace(/[^A-Za-z0-9_.]/g, "_"); const name = `@sc_arrops.${part(v.retain)}.${part(v.release)}.${part(trace)}`; host.declare( - `${name} = internal constant %ScrArrOps { ptr ${v.retain}, ptr ${v.release}, ptr ${trace} }`, + `${name} = internal global %ScrArrOps { ptr ${v.retain}, ptr ${v.release}, ptr ${trace}, i32 0 }`, ); return name; } diff --git a/packages/runtime/src/scr_array.c b/packages/runtime/src/scr_array.c index 4887f23ff..51853afce 100644 --- a/packages/runtime/src/scr_array.c +++ b/packages/runtime/src/scr_array.c @@ -42,14 +42,100 @@ static void scr_arr_oom(void) { static const ScrArrExt scr_arr_no_ext; SCR_TL bool (*scr_arr_match_copy)(ScrArr *a, ScrDyn *target) = NULL; -#define SCR_AX(a) ((const ScrArrExt *)((a)->ext ? (a)->ext : &scr_arr_no_ext)) +static inline const ScrArrExt *scr_arr_ax(const ScrArr *a) { + const ScrArrExt *x = scr_arr_ext_of(a); + return x ? x : &scr_arr_no_ext; +} +#define SCR_AX(a) scr_arr_ax(a) + +static void scr_arr_move_inline(ScrArr *a, size_t cap, size_t unit); + +/* Bytes before the dense slots' allocation start: the ext pointer's word. */ +static size_t scr_arr_prefix(const ScrArr *a) { + return (a->spare & SCR_ARR_HAS_EXT) ? SCR_ARR_EXT_PREFIX : 0; +} + +/* The start of the block the dense storage was allocated as. */ +static void *scr_arr_alloc_start(const ScrArr *a) { + return (char *)scr_arr_block(a) - scr_arr_prefix(a); +} + +/* Bytes per dense slot with its state: an 8-byte value, or a boolean byte. */ +static size_t scr_arr_slot_unit(const ScrArr *a) { + return a->elem == SCR_ELEM_BOOL ? 2 * sizeof(uint8_t) : sizeof(uint64_t) + sizeof(uint8_t); +} + +/* Bytes of dense storage (head room included) after the prefix. */ +static size_t scr_arr_storage_bytes(const ScrArr *a) { + return ((size_t)(a->data - scr_arr_block(a)) + a->cap) * scr_arr_slot_unit(a); +} ScrArrExt *scr_arr_ext(ScrArr *a) { - if (!a->ext) { - a->ext = scr_mem_calloc(sizeof(ScrArrExt)); - if (!a->ext) scr_arr_oom(); + ScrArrExt *x = scr_arr_ext_of(a); + if (x) return x; + x = scr_mem_calloc(sizeof(ScrArrExt)); + if (!x) scr_arr_oom(); + /* The pointer goes in the word before the dense allocation, which an + * array with ext always has: inline slots move out, and an array without + * storage gets an allocation of the word alone. */ + if (a->spare & SCR_ARR_INLINE_SLOTS) scr_arr_move_inline(a, a->cap, scr_arr_slot_unit(a)); + char *start; + if (!a->data) { + start = scr_mem_alloc(SCR_ARR_EXT_PREFIX); + if (!start) scr_arr_oom(); + a->data = (uint64_t *)(start + SCR_ARR_EXT_PREFIX); + a->cap = 0; + } else { + size_t bytes = scr_arr_storage_bytes(a); + size_t head = (size_t)(a->data - scr_arr_block(a)); + if (a->rc == SIZE_MAX) { + /* An immortal array's storage may be static: copy it, never move it. */ + start = scr_mem_alloc(bytes + SCR_ARR_EXT_PREFIX); + if (!start) scr_arr_oom(); + memcpy(start + SCR_ARR_EXT_PREFIX, scr_arr_block(a), bytes); + } else { + start = scr_mem_realloc(scr_arr_block(a), bytes + SCR_ARR_EXT_PREFIX); + if (!start) scr_arr_oom(); + memmove(start + SCR_ARR_EXT_PREFIX, start, bytes); + } + a->data = (uint64_t *)(start + SCR_ARR_EXT_PREFIX) + head; } - return a->ext; + memcpy(start, &x, sizeof x); + a->spare |= SCR_ARR_HAS_EXT; + return x; +} + +/* Registry of element ops by id (scr_arr_ops); slot 0 is no ops. */ +const ScrArrOps *scr_arr_ops_table[SCR_ARR_OPS_OVERFLOW] = {0}; +static uint32_t scr_arr_ops_count = 1; + +/* Assign `ops` its id on its first array; any thread may get there first, + * and every thread then sees the same id. A lost race leaves one table + * slot unused. Once the table is full, later descriptors get + * SCR_ARR_OPS_OVERFLOW. */ +uint32_t scr_arr_ops_register(ScrArrOps *ops) { + uint32_t id = __atomic_load_n(&ops->id, __ATOMIC_ACQUIRE); + if (id) return id; + uint32_t fresh = __atomic_fetch_add(&scr_arr_ops_count, 1, __ATOMIC_RELAXED); + if (fresh < SCR_ARR_OPS_OVERFLOW) { + __atomic_store_n(&scr_arr_ops_table[fresh], ops, __ATOMIC_RELEASE); + } else { + fresh = SCR_ARR_OPS_OVERFLOW; + } + uint32_t expected = 0; + if (__atomic_compare_exchange_n(&ops->id, &expected, fresh, false, __ATOMIC_ACQ_REL, + __ATOMIC_ACQUIRE)) + return fresh; + return expected; +} + +const ScrArrOps *scr_arr_ops_overflow(const ScrArr *a) { return scr_arr_ext_of(a)->ops; } + +/* Record `ops` as the element ops of `a` (NULL for none). */ +static void scr_arr_set_ops(ScrArr *a, const ScrArrOps *ops) { + uint32_t id = scr_arr_ops_id(ops); + a->spare = (uint16_t)((a->spare & ((1u << SCR_ARR_OPS_SHIFT) - 1)) | (id << SCR_ARR_OPS_SHIFT)); + if (id == SCR_ARR_OPS_OVERFLOW) scr_arr_ext(a)->ops = ops; } /* Canonical array indices grow indexed storage and may leave holes. Negative, @@ -133,24 +219,36 @@ static bool scr_elem_is_ref(ScrElemKind k) { k == SCR_ELEM_REF; } -static void scr_elem_release(const ScrArr *a, uint64_t slot) { +/* Releases an element slot of kind `elem`, with element ops `ops` for + * SCR_ELEM_REF. An array's kind and ops never change, so loops look them up + * once (element releases may run teardown, which reloads anything read + * through the array). */ +static inline void scr_elem_release_with(uint8_t elem, const ScrArrOps *ops, uint64_t slot) { void *p = scr_slot_to_ptr(slot); if (!p) return; - if (a->elem == SCR_ELEM_STR) scr_str_release((ScrStr *)p); - else if (a->elem == SCR_ELEM_ARR) scr_arr_release((ScrArr *)p); - else if (a->elem == SCR_ELEM_BYTES) scr_bytes_release((ScrBytes *)p); - else if (a->elem == SCR_ELEM_REF) a->ops->release(p); + if (elem == SCR_ELEM_STR) scr_str_release((ScrStr *)p); + else if (elem == SCR_ELEM_ARR) scr_arr_release((ScrArr *)p); + else if (elem == SCR_ELEM_BYTES) scr_bytes_release((ScrBytes *)p); + else if (elem == SCR_ELEM_REF) ops->release(p); +} + +static inline uint64_t scr_elem_retain_with(uint8_t elem, const ScrArrOps *ops, uint64_t slot) { + if (!scr_elem_is_ref(elem)) return slot; + void *p = scr_slot_to_ptr(slot); + if (!p) return slot; + if (elem == SCR_ELEM_STR) p = scr_str_retain((ScrStr *)p); + else if (elem == SCR_ELEM_ARR) p = scr_arr_retain((ScrArr *)p); + else if (elem == SCR_ELEM_BYTES) p = scr_bytes_retain((ScrBytes *)p); + else p = ops->retain(p); + return scr_slot_from_ptr(p); +} + +static void scr_elem_release(const ScrArr *a, uint64_t slot) { + scr_elem_release_with(a->elem, scr_arr_ops(a), slot); } static uint64_t scr_elem_retain_slot(const ScrArr *a, uint64_t slot) { - if (!scr_elem_is_ref(a->elem)) return slot; - void *p = scr_slot_to_ptr(slot); - if (!p) return slot; - if (a->elem == SCR_ELEM_STR) p = scr_str_retain((ScrStr *)p); - else if (a->elem == SCR_ELEM_ARR) p = scr_arr_retain((ScrArr *)p); - else if (a->elem == SCR_ELEM_BYTES) p = scr_bytes_retain((ScrBytes *)p); - else p = a->ops->retain(p); - return scr_slot_from_ptr(p); + return scr_elem_retain_with(a->elem, scr_arr_ops(a), slot); } static uint64_t scr_arr_retain_state_slot(const ScrArr *a, uint64_t slot, @@ -310,7 +408,6 @@ static void scr_arr_sparse_drop_prefix(ScrArr *a, size_t count) { * `spare` records it. Growth past the inline capacity moves the slots to an * ordinary allocation; the inline room then stays unused until the array is * freed. */ -#define SCR_ARR_INLINE_SLOTS 1u /* Where inline slots start in the array's block (8-byte aligned on every * target). */ #define SCR_ARR_INLINE_OFFSET ((sizeof(ScrArr) + 7) & ~(size_t)7) @@ -329,11 +426,10 @@ static bool scr_arr_inline(const ScrArr *a) { return (a->spare & SCR_ARR_INLINE_ * slot just before `data` holds its size (scr_arr_head). Every reader * indexes from `data` and `cap` as before (emitted code included); only the * code that frees, reallocates or swaps the allocation needs its start. - * Static and inline storage never has head room. */ -#define SCR_ARR_HEAD_ROOM 2u -/* An unshift had to move every element: the next one that would have to - * move them again opens head room instead (scr_arr_open_head). */ -#define SCR_ARR_UNSHIFTED 4u + * Static and inline storage never has head room. SCR_ARR_UNSHIFTED in + * `spare` records that an unshift had to move every element, so the next + * one that would have to move them again opens head room instead + * (scr_arr_open_head). */ /* Arrays shorter than this keep moving their elements on unshift. */ #define SCR_ARR_HEAD_MIN ((size_t)16) @@ -351,8 +447,6 @@ static void scr_arr_set_head(ScrArr *a, size_t head) { a->spare |= SCR_ARR_HEAD_ROOM; } -/* The start of the dense allocation. */ -static void *scr_arr_block(const ScrArr *a) { return a->data - scr_arr_head(a); } /* Give the head room of an array back to its capacity: the populated dense * prefix and its states move to the start of the allocation. */ @@ -404,8 +498,8 @@ static bool scr_arr_drop_front(ScrArr *a, size_t count) { } /* Slots requested by the small constructions. The small allocator rounds - * them up to fill its block: 3 eight-byte slots after a plain header in an - * 80-byte block (88 after a collector header), or 8 booleans in a 64-byte + * them up to fill its block: 3 eight-byte slots after a plain header in a + * 64-byte block (72 after a collector header), or 4 booleans in a 40-byte * block. Most small arrays hold up to three elements; two slots make * arrays that grow past them move their storage often enough to slow * programs that build many short lists. */ @@ -497,8 +591,8 @@ static void scr_arr_grow_dense_slow(ScrArr *a, size_t need, ScrArrGrow mode); * paths pick the same capacity. */ static inline void scr_arr_grow_dense_to(ScrArr *a, size_t need, ScrArrGrow mode) { if (need <= a->cap) return; - if (a->elem == SCR_ELEM_BOOL || (a->spare & (SCR_ARR_INLINE_SLOTS | SCR_ARR_HEAD_ROOM)) || - (a->ext && a->ext->sparse_len) || need > SCR_ARR_DENSE_LIMIT) { + if (a->elem == SCR_ELEM_BOOL || (a->spare & (SCR_ARR_INLINE_SLOTS | SCR_ARR_HEAD_ROOM | SCR_ARR_HAS_EXT)) || + need > SCR_ARR_DENSE_LIMIT) { scr_arr_grow_dense_slow(a, need, mode); return; } @@ -524,6 +618,7 @@ static inline void scr_arr_grow_dense_to(ScrArr *a, size_t need, ScrArrGrow mode /* Sparse entries below a raised capacity move into the packed slots. */ static void scr_arr_absorb_sparse(ScrArr *a) { + if (!(a->spare & SCR_ARR_HAS_EXT)) return; size_t moved = scr_arr_sparse_lower_bound(SCR_AX(a)->sparse, SCR_AX(a)->sparse_len, a->cap); for (size_t i = 0; i < moved; i++) { scr_arr_dense_put(a, SCR_AX(a)->sparse[i].index, SCR_AX(a)->sparse[i].slot); @@ -585,8 +680,10 @@ static void scr_arr_grow_dense_slow(ScrArr *a, size_t need, ScrArrGrow mode) { /* Values and states share one allocation. Move the old states to its new * end before publishing the larger capacity: their previous location is * now part of the value area. */ - uint64_t *data = scr_mem_realloc(a->data, cap * unit); - if (!data) scr_arr_oom(); + size_t prefix = scr_arr_prefix(a); + char *start = scr_mem_realloc((char *)a->data - prefix, prefix + cap * unit); + if (!start) scr_arr_oom(); + uint64_t *data = (uint64_t *)(start + prefix); uint8_t *end = (uint8_t *)data + cap * (width + 1); if (a->cap) memmove(end - a->cap, (uint8_t *)data + a->cap * width, a->cap); memset(end - cap, SCR_ARR_HOLE, cap - a->cap); @@ -621,8 +718,12 @@ static bool scr_arr_open_head(ScrArr *a, size_t count) { size_t room = count + (len + count) / 2; if (room > SCR_ARR_MAX_LENGTH || a->cap > SIZE_MAX / SCR_ARR_SLOT_BYTES - room) return false; size_t total = room + a->cap; - uint64_t *block = scr_mem_alloc(total * SCR_ARR_SLOT_BYTES); - if (!block) scr_arr_oom(); + size_t prefix = scr_arr_prefix(a); + char *start = scr_mem_alloc(prefix + total * SCR_ARR_SLOT_BYTES); + if (!start) scr_arr_oom(); + /* The ext pointer's word moves with the allocation. */ + if (prefix) memcpy(start, scr_arr_alloc_start(a), prefix); + uint64_t *block = (uint64_t *)(start + prefix); /* The window's states end `room` bytes before the allocation does; the * room's states past it are holes. */ uint8_t *end = (uint8_t *)(block + total) + a->cap; @@ -630,7 +731,7 @@ static bool scr_arr_open_head(ScrArr *a, size_t count) { memset(end, SCR_ARR_HOLE, room); memcpy(end - len, scr_arr_state_run(a, 0, len), len); memset(end - a->cap, SCR_ARR_HOLE, a->cap - len); - scr_mem_free(scr_arr_block(a)); + scr_mem_free(scr_arr_alloc_start(a)); a->data = block + room; head = room; } @@ -700,21 +801,40 @@ static uint8_t scr_arr_state_at_storage(const ScrArrStorage *s, size_t index, return s->sparse[pos].state; } -static bool scr_arr_slot_at_storage(const ScrArrStorage *s, size_t index, - uint64_t *out) { - return scr_arr_state_at_storage(s, index, out) != SCR_ARR_HOLE; +/* A state and its slot (zero for a hole). */ +typedef struct { + uint64_t slot; + uint8_t state; +} ScrArrStateSlot; + +/* scr_arr_state_at past the dense capacity: the sorted sparse entries. It + * only reads memory, so callers keep the array's fields in registers across + * it. */ +static __attribute__((noinline, pure)) ScrArrStateSlot scr_arr_sparse_state_at(const ScrArr *a, + size_t index) { + const ScrArrExt *x = SCR_AX(a); + size_t pos = scr_arr_sparse_lower_bound(x->sparse, x->sparse_len, index); + if (pos == x->sparse_len || x->sparse[pos].index != index) return (ScrArrStateSlot){0, SCR_ARR_HOLE}; + return (ScrArrStateSlot){x->sparse[pos].slot, x->sparse[pos].state}; } -static bool scr_arr_slot_at(const ScrArr *a, size_t index, uint64_t *out) { - ScrArrStorage s = {a->data, scr_arr_states_end(a), a->cap, SCR_AX(a)->sparse, - SCR_AX(a)->sparse_len, SCR_AX(a)->sparse_cap, a->elem, NULL}; - return scr_arr_slot_at_storage(&s, index, out); +/* The state at `index` and, unless it is a hole, its slot in `out`: the + * dense storage below `cap` (without touching the out-of-line storage), the + * sparse entries above it. */ +static inline uint8_t scr_arr_state_at(const ScrArr *a, size_t index, uint64_t *out) { + if (index < a->cap) { + uint8_t state = *scr_arr_state_in(scr_arr_states_end(a), index); + if (state == SCR_ARR_HOLE) return SCR_ARR_HOLE; + if (out) *out = scr_arr_dense_get(a, index); + return state; + } + ScrArrStateSlot sparse = scr_arr_sparse_state_at(a, index); + if (sparse.state != SCR_ARR_HOLE && out) *out = sparse.slot; + return sparse.state; } -static uint8_t scr_arr_state_at(const ScrArr *a, size_t index, uint64_t *out) { - ScrArrStorage s = {a->data, scr_arr_states_end(a), a->cap, SCR_AX(a)->sparse, - SCR_AX(a)->sparse_len, SCR_AX(a)->sparse_cap, a->elem, NULL}; - return scr_arr_state_at_storage(&s, index, out); +static inline bool scr_arr_slot_at(const ScrArr *a, size_t index, uint64_t *out) { + return scr_arr_state_at(a, index, out) != SCR_ARR_HOLE; } /* Store an owned slot into an empty index. The caller has already established @@ -843,7 +963,7 @@ static void scr_arr_prop_set(ScrArr *a, double key, uint64_t slot) { char *copy = malloc(len + 1); if (!copy) scr_arr_oom(); memcpy(copy, text, len + 1); - a->ext->props[a->ext->prop_len++] = (ScrArrProp){copy, len, slot, SCR_ARR_VALUE}; + scr_arr_ext_of(a)->props[scr_arr_ext_of(a)->prop_len++] = (ScrArrProp){copy, len, slot, SCR_ARR_VALUE}; } static void scr_arr_prop_set_undefined(ScrArr *a, double key) { @@ -864,7 +984,7 @@ static void scr_arr_prop_set_undefined(ScrArr *a, double key) { char *copy = malloc(len + 1); if (!copy) scr_arr_oom(); memcpy(copy, text, len + 1); - a->ext->props[a->ext->prop_len++] = (ScrArrProp){copy, len, 0, SCR_ARR_UNDEFINED}; + scr_arr_ext_of(a)->props[scr_arr_ext_of(a)->prop_len++] = (ScrArrProp){copy, len, 0, SCR_ARR_UNDEFINED}; } static bool scr_arr_prop_delete(ScrArr *a, double key) { @@ -877,7 +997,7 @@ static bool scr_arr_prop_delete(ScrArr *a, double key) { free(SCR_AX(a)->props[pos].key); memmove(SCR_AX(a)->props + pos, SCR_AX(a)->props + pos + 1, (SCR_AX(a)->prop_len - pos - 1) * sizeof(*SCR_AX(a)->props)); - a->ext->prop_len--; + scr_arr_ext_of(a)->prop_len--; if (old_state == SCR_ARR_VALUE && scr_elem_is_ref(a->elem)) { scr_elem_release(a, old); } @@ -885,17 +1005,25 @@ static bool scr_arr_prop_delete(ScrArr *a, double key) { } static ScrArrStorage scr_arr_take_storage(ScrArr *a) { + ScrArrExt *x = scr_arr_ext_of(a); ScrArrStorage old = {a->data, scr_arr_states_end(a), a->cap, SCR_AX(a)->sparse, SCR_AX(a)->sparse_len, SCR_AX(a)->sparse_cap, a->elem, - scr_arr_inline(a) ? NULL : scr_arr_block(a)}; - a->spare &= (uint16_t)~(SCR_ARR_INLINE_SLOTS | SCR_ARR_HEAD_ROOM); + scr_arr_inline(a) ? NULL : scr_arr_alloc_start(a)}; + a->spare &= (uint16_t)~(SCR_ARR_INLINE_SLOTS | SCR_ARR_HEAD_ROOM | SCR_ARR_HAS_EXT); a->data = NULL; a->cap = 0; a->filled = 0; - if (a->ext) { - a->ext->sparse = NULL; - a->ext->sparse_len = 0; - a->ext->sparse_cap = 0; + if (x) { + /* The ext pointer's word left with the old allocation: the array keeps + * its ext in an allocation of the word alone. */ + char *start = scr_mem_alloc(SCR_ARR_EXT_PREFIX); + if (!start) scr_arr_oom(); + memcpy(start, &x, sizeof x); + a->data = (uint64_t *)(start + SCR_ARR_EXT_PREFIX); + a->spare |= SCR_ARR_HAS_EXT; + x->sparse = NULL; + x->sparse_len = 0; + x->sparse_cap = 0; } return old; } @@ -910,7 +1038,7 @@ static void scr_arr_free_storage(ScrArrStorage *s) { * operations move ownership with the slots and move the states alongside * them; they never inspect an uninitialized hole payload. */ static bool scr_arr_is_dense(const ScrArr *a) { - return a->len <= a->cap && SCR_AX(a)->sparse_len == 0; + return a->len <= a->cap && (!(a->spare & SCR_ARR_HAS_EXT) || scr_arr_ext_of(a)->sparse_len == 0); } static void scr_arr_move_dense(ScrArr *a, size_t to, size_t from, size_t count) { @@ -1025,9 +1153,11 @@ static void scr_arr_copy_dense(ScrArr *dst, size_t to, const ScrArr *src, } bool refs = scr_elem_is_ref(src->elem); if (!refs && !materialize_holes) return; + uint8_t elem = src->elem; + const ScrArrOps *ops = scr_arr_ops(src); for (size_t i = to; i < to + count; i++) { if (*scr_arr_state_in(dst_states, i) == SCR_ARR_VALUE) { - if (refs) dst->data[i] = scr_elem_retain_slot(src, dst->data[i]); + if (refs) dst->data[i] = scr_elem_retain_with(elem, ops, dst->data[i]); } else if (materialize_holes && *scr_arr_state_in(dst_states, i) == SCR_ARR_HOLE) { scr_arr_dense_put(dst, i, 0); *scr_arr_state_in(dst_states, i) = SCR_ARR_UNDEFINED; @@ -1060,9 +1190,7 @@ static inline __attribute__((always_inline)) ScrArr *scr_arr_make(ScrElemKind el a->cyc = 0; a->spare = 0; a->filled = 0; - a->ops = NULL; a->data = NULL; - a->ext = NULL; if (inline_cap) scr_arr_use_inline(a, inline_cap); else if (initial_cap > 0) scr_arr_grow_dense_to(a, initial_cap, SCR_ARR_GROW_EXACT); #ifdef SCR_RC_AUDIT @@ -1094,9 +1222,7 @@ ScrArr *scr_arr_new_inline(ScrElemKind elem, size_t cap) { a->cyc = 0; a->spare = 0; a->filled = 0; - a->ops = NULL; a->data = NULL; - a->ext = NULL; scr_arr_use_inline(a, cap); #ifdef SCR_RC_AUDIT scr_live_arrays++; @@ -1159,20 +1285,25 @@ void scr_arr_trace_v(void *a0, ScrTraceVisit visit, void *ctx) { } /* Frees the out-of-line storage; callers check for it first, since most - * arrays have none. */ + * arrays have none. The word that pointed at it stays in the dense + * allocation until that is freed. */ static void scr_arr_ext_free(ScrArr *a) { - ScrArrExt *x = a->ext; + ScrArrExt *x = scr_arr_ext_of(a); scr_arr_sparse_free(x->sparse); for (size_t i = 0; i < x->prop_len; i++) free(x->props[i].key); free(x->props); scr_mem_free(x); - a->ext = NULL; +} + +/* Frees the dense allocation and the out-of-line storage of a dying array. */ +static inline __attribute__((always_inline)) void scr_arr_free_parts(ScrArr *a) { + if (a->spare & SCR_ARR_HAS_EXT) scr_arr_ext_free(a); + if (!scr_arr_inline(a)) scr_mem_free(scr_arr_alloc_start(a)); } static void scr_arr_gc_free(void *a0) { ScrArr *a = (ScrArr *)a0; - if (!scr_arr_inline(a)) scr_mem_free(scr_arr_block(a)); - if (a->ext) scr_arr_ext_free(a); + scr_arr_free_parts(a); #ifdef SCR_RC_AUDIT scr_live_arrays--; #endif @@ -1199,9 +1330,8 @@ static ScrArr *scr_arr_make_ops(const ScrArrOps *ops, size_t initial_cap, bool s a->cyc = cyc; a->spare = 0; a->filled = 0; - a->ops = ops; a->data = NULL; - a->ext = NULL; + scr_arr_set_ops(a, ops); if (inline_cap) scr_arr_use_inline(a, inline_cap); else if (initial_cap > 0) scr_arr_grow_dense_to(a, initial_cap, SCR_ARR_GROW_EXACT); #ifdef SCR_RC_AUDIT @@ -1218,9 +1348,9 @@ ScrArr *scr_arr_new_ops_small(const ScrArrOps *ops) { return scr_arr_make_ops(op static void scr_arr_destroy(void *object) { ScrArr *a = object; - if (a->ext) { - scr_dyn_release(a->ext->metadata); - a->ext->metadata = NULL; + if (scr_arr_ext_of(a)) { + scr_dyn_release(scr_arr_ext_of(a)->metadata); + scr_arr_ext_of(a)->metadata = NULL; } if (scr_elem_is_ref(a->elem)) { size_t end = a->len < a->cap ? a->len : a->cap; @@ -1243,30 +1373,32 @@ static void scr_arr_destroy(void *object) { } } else { uint8_t *states = scr_arr_states_end(a); + uint8_t elem = a->elem; + const ScrArrOps *ops = scr_arr_ops(a); for (size_t i = 0; i < end; i++) { if (*scr_arr_state_in(states, i) != SCR_ARR_HOLE) { uint64_t old = a->data[i]; uint8_t state = *scr_arr_state_in(states, i); *scr_arr_state_in(states, i) = 0; a->data[i] = 0; - if (state == SCR_ARR_VALUE) scr_elem_release(a, old); + if (state == SCR_ARR_VALUE) scr_elem_release_with(elem, ops, old); } } } - if (a->ext) { - while (a->ext->sparse_len > 0) { - size_t i = a->ext->sparse_len - 1; - uint64_t old = a->ext->sparse[i].slot; - uint8_t state = a->ext->sparse[i].state; - a->ext->sparse_len = i; + if (scr_arr_ext_of(a)) { + while (scr_arr_ext_of(a)->sparse_len > 0) { + size_t i = scr_arr_ext_of(a)->sparse_len - 1; + uint64_t old = scr_arr_ext_of(a)->sparse[i].slot; + uint8_t state = scr_arr_ext_of(a)->sparse[i].state; + scr_arr_ext_of(a)->sparse_len = i; if (state == SCR_ARR_VALUE) scr_elem_release(a, old); } - while (a->ext->prop_len > 0) { - size_t i = a->ext->prop_len - 1; - uint64_t old = a->ext->props[i].slot; - uint8_t state = a->ext->props[i].state; - free(a->ext->props[i].key); - a->ext->prop_len = i; + while (scr_arr_ext_of(a)->prop_len > 0) { + size_t i = scr_arr_ext_of(a)->prop_len - 1; + uint64_t old = scr_arr_ext_of(a)->props[i].slot; + uint8_t state = scr_arr_ext_of(a)->props[i].state; + free(scr_arr_ext_of(a)->props[i].key); + scr_arr_ext_of(a)->prop_len = i; if (state == SCR_ARR_VALUE) scr_elem_release(a, old); } } @@ -1274,8 +1406,7 @@ static void scr_arr_destroy(void *object) { if (a->cyc) { scr_arr_gc_free(a); } else { - if (!scr_arr_inline(a)) scr_mem_free(scr_arr_block(a)); - if (a->ext) scr_arr_ext_free(a); + scr_arr_free_parts(a); #ifdef SCR_RC_AUDIT scr_live_arrays--; #endif @@ -1303,7 +1434,10 @@ bool scr_arr_has(const ScrArr *a, double i) { return scr_arr_slot_at(a, idx, NULL); } -double scr_arr_state(const ScrArr *a, double i) { +/* Emitted element reads call this on their cold path. It stays out of line + * so that, under link-time optimization, those reads stay small enough to + * inline where they are used. */ +__attribute__((noinline)) double scr_arr_state(const ScrArr *a, double i) { size_t idx; if (!scr_arr_valid_index(i, &idx)) { uint8_t state; @@ -1359,13 +1493,13 @@ void scr_arr_set_len(ScrArr *a, double length) { if (state == SCR_ARR_VALUE) scr_elem_release(a, old); } } - if (a->ext) { - size_t keep = scr_arr_sparse_lower_bound(a->ext->sparse, a->ext->sparse_len, next); - while (a->ext->sparse_len > keep) { - size_t i = a->ext->sparse_len - 1; - uint64_t old = a->ext->sparse[i].slot; - uint8_t state = a->ext->sparse[i].state; - a->ext->sparse_len = i; + if (scr_arr_ext_of(a)) { + size_t keep = scr_arr_sparse_lower_bound(scr_arr_ext_of(a)->sparse, scr_arr_ext_of(a)->sparse_len, next); + while (scr_arr_ext_of(a)->sparse_len > keep) { + size_t i = scr_arr_ext_of(a)->sparse_len - 1; + uint64_t old = scr_arr_ext_of(a)->sparse[i].slot; + uint8_t state = scr_arr_ext_of(a)->sparse[i].state; + scr_arr_ext_of(a)->sparse_len = i; if (state == SCR_ARR_VALUE) scr_elem_release(a, old); } } @@ -1375,7 +1509,7 @@ void scr_arr_set_len(ScrArr *a, double length) { memset(scr_arr_dense_at(a, dense_stop), 0, (dense_end - dense_stop) * scr_arr_slot_width(a)); } - if (a->ext) a->ext->sparse_len = scr_arr_sparse_lower_bound(a->ext->sparse, a->ext->sparse_len, next); + if (scr_arr_ext_of(a)) scr_arr_ext_of(a)->sparse_len = scr_arr_sparse_lower_bound(scr_arr_ext_of(a)->sparse, scr_arr_ext_of(a)->sparse_len, next); } } a->len = next; @@ -1709,7 +1843,7 @@ void *scr_arr_get_ref(ScrArr *a, double i) { if (!p) return NULL; if (a->elem == SCR_ELEM_STR) scr_str_retain((ScrStr *)p); else if (a->elem == SCR_ELEM_BYTES) scr_bytes_retain((ScrBytes *)p); - else if (a->elem == SCR_ELEM_REF) p = a->ops->retain(p); + else if (a->elem == SCR_ELEM_REF) p = scr_arr_ops(a)->retain(p); else scr_arr_retain((ScrArr *)p); return p; } @@ -2193,7 +2327,7 @@ ScrArr *scr_arr_splice(ScrArr *a, double start, double deleteCount) { size_t hint = scr_arr_is_dense(a) || n <= SCR_ARR_DENSE_LIMIT ? n : SCR_ARR_DENSE_LIMIT; ScrArr *out = a->elem == SCR_ELEM_REF - ? scr_arr_new_ops(a->ops, hint ? hint : 1) + ? scr_arr_new_ops(scr_arr_ops(a), hint ? hint : 1) : scr_arr_new(a->elem, hint ? hint : 1); out->len = n; size_t old_len = a->len; @@ -2273,7 +2407,7 @@ ScrArr *scr_arr_splice_insert(ScrArr *a, double start, double deleteCount, ScrArr *snapshot = items == a ? scr_arr_slice(a, 0, INFINITY) : NULL; if (snapshot) items = snapshot; ScrArr *removed = a->elem == SCR_ELEM_REF - ? scr_arr_new_ops(a->ops, n) + ? scr_arr_new_ops(scr_arr_ops(a), n) : scr_arr_new(a->elem, n); if (n) { memcpy(removed->data, scr_arr_dense_at(a, from), n * scr_arr_slot_width(a)); @@ -2319,7 +2453,7 @@ void scr_arr_splice_drop(ScrArr *a, double start, double deleteCount, size_t cou bool refs = scr_elem_is_ref(a->elem); if (!scr_arr_is_dense(a) || !scr_arr_dense_ok(a, next_len, count)) { ScrArr *items = a->elem == SCR_ELEM_REF - ? scr_arr_new_ops(a->ops, count) + ? scr_arr_new_ops(scr_arr_ops(a), count) : scr_arr_new(a->elem, count); scr_arr_push_many(items, count, slots); scr_arr_release(scr_arr_splice_insert(a, start, deleteCount, items)); @@ -2400,10 +2534,10 @@ ScrArr *scr_arr_flat_copy(const ScrArr *a, ScrArr *out, bool flatten) { static bool scr_arr_ref_eq(const ScrArr *a, uint64_t slot, void *v) { void *p = scr_slot_to_ptr(slot); if (a->elem == SCR_ELEM_STR) return scr_str_eq((ScrStr *)p, (ScrStr *)v); - if (a->elem == SCR_ELEM_REF && a->ops->retain == scr_dyn_retain_v) { + if (a->elem == SCR_ELEM_REF && scr_arr_ops(a)->retain == scr_dyn_retain_v) { return scr_dyn_strict_eq((ScrDyn *)p, (ScrDyn *)v); } - if (a->elem == SCR_ELEM_REF && a->ops->retain == scr_closure_retain_v) + if (a->elem == SCR_ELEM_REF && scr_arr_ops(a)->retain == scr_closure_retain_v) return scr_closure_identity_equal((ScrClosure *)p, (ScrClosure *)v); return p == v; } @@ -2461,7 +2595,7 @@ bool scr_arr_includes_bool(ScrArr *a, bool v) { } bool scr_arr_includes_ref(ScrArr *a, void *v) { - if (a->elem == SCR_ELEM_REF && a->ops->retain == scr_dyn_retain_v) { + if (a->elem == SCR_ELEM_REF && scr_arr_ops(a)->retain == scr_dyn_retain_v) { const ScrDyn *needle = (const ScrDyn *)v; if (needle->kind == SCR_DYN_NUM && needle->v.num != needle->v.num) { const uint8_t *states = scr_arr_states_end(a); @@ -2525,7 +2659,7 @@ ScrArr *scr_arr_slice(ScrArr *a, double start, double end) { size_t hint = to <= a->cap || n <= SCR_ARR_DENSE_LIMIT ? n : SCR_ARR_DENSE_LIMIT; ScrArr *out = a->elem == SCR_ELEM_REF - ? scr_arr_new_ops(a->ops, hint ? hint : 1) + ? scr_arr_new_ops(scr_arr_ops(a), hint ? hint : 1) : scr_arr_new(a->elem, hint ? hint : 1); out->len = n; if (to <= a->cap && to <= a->filled && a->elem != SCR_ELEM_BOOL && !scr_elem_is_ref(a->elem)) { @@ -2828,7 +2962,7 @@ static ScrSortCompare scr_sort_compare_for(const ScrArr *a, ScrClosure *f, uint3 if (a->elem == SCR_ELEM_STR) c.retain = scr_sort_retain_str; else if (a->elem == SCR_ELEM_ARR) c.retain = scr_sort_retain_arr; else if (a->elem == SCR_ELEM_BYTES) c.retain = scr_sort_retain_bytes; - else if (a->elem == SCR_ELEM_REF) c.retain = a->ops->retain; + else if (a->elem == SCR_ELEM_REF) c.retain = scr_arr_ops(a)->retain; else scr_trap("scriptc: invalid comparator sort element kind\n"); } return c; @@ -4441,6 +4575,8 @@ const uint64_t *scr_arr_dense_slots(const ScrArr *a, size_t *len) { * slots of `a` in place. */ bool scr_arr_sort_dense(ScrArr *a, ScrClosure *f, uint32_t arity, bool owned, int32_t order, ScrClosure *key) { + uint8_t elem = a->elem; + const ScrArrOps *ops = scr_arr_ops(a); if (a->rc == SIZE_MAX || !scr_arr_dense_values(a)) return false; size_t n = a->len; if (n < 2) return true; @@ -4459,10 +4595,10 @@ bool scr_arr_sort_dense(ScrArr *a, ScrClosure *f, uint32_t arity, bool owned, in if (!src) scr_arr_oom(); memcpy(src, a->data, n * sizeof(*src)); if (retained) - for (size_t i = 0; i < n; i++) src[i] = scr_elem_retain_slot(a, src[i]); + for (size_t i = 0; i < n; i++) src[i] = scr_elem_retain_with(elem, ops, src[i]); if (!scr_sort_slots(src, n, &c, true)) { if (retained) - for (size_t i = 0; i < n; i++) scr_elem_release(a, src[i]); + for (size_t i = 0; i < n; i++) scr_elem_release_with(elem, ops, src[i]); free(src); return true; } @@ -4470,7 +4606,7 @@ bool scr_arr_sort_dense(ScrArr *a, ScrClosure *f, uint32_t arity, bool owned, in /* The comparator published the array: the first write fails as in Node * (scr_throw_published_array's index write). */ if (retained) - for (size_t i = 0; i < n; i++) scr_elem_release(a, src[i]); + for (size_t i = 0; i < n; i++) scr_elem_release_with(elem, ops, src[i]); static const char assign[] = "Cannot assign to read only property '0' of object '[object Array]'"; static const char add[] = "Cannot add property 0, object is not extensible"; if (a->len > 0) scr_throw_error_msg(SCR_ERR_TYPE, assign, sizeof assign - 1); diff --git a/packages/runtime/src/scr_array.h b/packages/runtime/src/scr_array.h index 1678d1568..95ee7ce00 100644 --- a/packages/runtime/src/scr_array.h +++ b/packages/runtime/src/scr_array.h @@ -60,6 +60,9 @@ typedef struct ScrArrOps { void *(*retain)(void *); void (*release)(void *); ScrTraceFn trace; + /* Registry id (scr_arr_ops_table), 0 until scr_arr_ops_register assigns + * it on the first array of the type. Descriptors are writable for it. */ + uint32_t id; } ScrArrOps; typedef struct ScrArr { @@ -78,7 +81,10 @@ typedef struct ScrArr { * ref elements and the historic element kinds keep the lean 1-word * header. Kept beside `elem` so the RC fast paths test it with one load. */ uint8_t cyc; - /* Storage flags private to scr_array.c (inline slots, head room). */ + /* The low SCR_ARR_OPS_SHIFT bits: storage flags (SCR_ARR_INLINE_SLOTS, + * SCR_ARR_HEAD_ROOM, SCR_ARR_UNSHIFTED, SCR_ARR_HAS_EXT). The rest: the + * registry id of the element type's ops (scr_arr_ops), 0 for every kind + * but SCR_ELEM_REF. */ uint16_t spare; /* A lower bound on the hole-free prefix: filled <= len, filled <= cap, * and every dense state below it is SCR_ARR_VALUE. Element reads below @@ -87,17 +93,15 @@ typedef struct ScrArr { * undefined) or move states below it lowers it first (scr_arr_cut); * 0 is always valid. */ uint32_t filled; - /* SCR_ELEM_REF only; NULL for every other element kind. */ - const ScrArrOps *ops; /* Dense values: `cap` 8-byte slots, or `cap` bytes for SCR_ELEM_BOOL * (scr_arr_slot_width), followed in the same allocation by one state byte * per slot in reverse slot order (scr_arr_states_end). Sparse entries and properties keep 8-byte - * slots for every kind; a bool slot is 0 or 1 wherever it lives. */ + * slots for every kind; a bool slot is 0 or 1 wherever it lives. Rarely + * used storage (sparse indices, noncanonical numeric properties, + * match-result metadata) lives out of line, and an array that has it + * (SCR_ARR_HAS_EXT) keeps a pointer to it in the word before the dense + * allocation (scr_arr_ext_of). */ uint64_t *data; - /* Rarely used storage (sparse indices, noncanonical numeric properties, - * match-result metadata) lives out of line: NULL until first needed, so - * the common dense array stays small. */ - struct ScrArrExt *ext; } ScrArr; typedef struct ScrArrExt { @@ -113,11 +117,14 @@ typedef struct ScrArrExt { * scr_regex.c), and match_index is the result's UTF-16 start index. */ struct ScrDyn *metadata; double match_index; + /* The element ops of an array whose ops have no registry id + * (SCR_ARR_OPS_OVERFLOW). */ + const ScrArrOps *ops; } ScrArrExt; /* The LLVM backend mirrors ScrArr as %ScrArr: its inline retain/release read * `cyc` (field 4, i8) to decide mark-live / candidate buffering exactly like * scr_arr_retain and scr_arr_release; its inline element access uses len and - * cap (fields 1 and 2, i32), filled (field 6, i32) and data (field 8), and + * cap (fields 1 and 2, i32), filled (field 6, i32) and data (field 7), and * finds the state of slot i at data + cap * (slot width + 1) - 1 - i. */ _Static_assert(offsetof(ScrArr, len) == sizeof(size_t) && sizeof(((ScrArr *)0)->len) == 4, "emitted length reads load ScrArr.len as %ScrArr field 1, an i32"); @@ -127,10 +134,68 @@ _Static_assert(offsetof(ScrArr, cyc) == sizeof(size_t) + 9, "inline RC fast paths read ScrArr.cyc as %ScrArr field 4"); _Static_assert(offsetof(ScrArr, filled) == sizeof(size_t) + 12, "inline element access reads ScrArr.filled as %ScrArr field 6"); -_Static_assert(offsetof(ScrArr, data) == sizeof(size_t) + 16 + sizeof(void *), - "inline element reads use ScrArr.data as %ScrArr field 8"); -_Static_assert(sizeof(ScrArr) == sizeof(size_t) + 16 + 3 * sizeof(void *), - "ScrArr is 48 bytes on 64-bit targets"); +_Static_assert(offsetof(ScrArr, data) == sizeof(size_t) + 16, + "inline element reads use ScrArr.data as %ScrArr field 7"); +_Static_assert(sizeof(ScrArr) == sizeof(size_t) + 16 + sizeof(void *), + "ScrArr is 32 bytes on 64-bit targets"); + +/* Storage flags in `spare`. Inline slots: the dense slots live in the + * array's own block, after the header. Head room: `data` has advanced past + * vacated slots of its allocation, and the slot before `data` holds their + * count. Unshifted: an unshift had to move every element. Has ext: the word + * before the dense allocation (head room included) points at the array's + * ScrArrExt; such an array always has an allocation of its own, without + * inline slots. */ +#define SCR_ARR_INLINE_SLOTS 1u +#define SCR_ARR_HEAD_ROOM 2u +#define SCR_ARR_UNSHIFTED 4u +#define SCR_ARR_HAS_EXT 8u + +/* The start of the dense slots' allocation: `data` less any head room. */ +static inline uint64_t *scr_arr_block(const ScrArr *a) { + return (a->spare & SCR_ARR_HEAD_ROOM) ? a->data - (size_t)a->data[-1] : a->data; +} + +/* Bytes before the dense allocation of an array with out-of-line storage: + * its pointer, in a word that keeps the 8-byte slots aligned on every + * target. */ +#define SCR_ARR_EXT_PREFIX sizeof(uint64_t) + +/* The out-of-line storage of `a`, or NULL while it has none. */ +static inline struct ScrArrExt *scr_arr_ext_of(const ScrArr *a) { + if (!(a->spare & SCR_ARR_HAS_EXT)) return NULL; + struct ScrArrExt *x; + memcpy(&x, (const char *)scr_arr_block(a) - SCR_ARR_EXT_PREFIX, sizeof x); + return x; +} + +/* Element ops by registry id: `spare >> SCR_ARR_OPS_SHIFT` of an array. + * Id 0 is no ops; ids are assigned on each descriptor's first array, in + * any thread. A program with more element types than the field holds keeps + * the ops of the rest in the array's ext (SCR_ARR_OPS_OVERFLOW). */ +#define SCR_ARR_OPS_SHIFT 4 +#define SCR_ARR_OPS_OVERFLOW (UINT16_MAX >> SCR_ARR_OPS_SHIFT) +extern const ScrArrOps *scr_arr_ops_table[SCR_ARR_OPS_OVERFLOW]; +/* Assigns the id on a descriptor's first array; out of line, so the id + * load in scr_arr_ops_id stays the only cost on later arrays. */ +__attribute__((cold, noinline)) uint32_t scr_arr_ops_register(ScrArrOps *ops); + +/* The registry id an array of element ops `ops` stores (0 for NULL). */ +static inline uint32_t scr_arr_ops_id(const ScrArrOps *ops) { + if (!ops) return 0; + uint32_t id = __atomic_load_n(&ops->id, __ATOMIC_ACQUIRE); + return id ? id : scr_arr_ops_register((ScrArrOps *)ops); +} + +/* The element ops of an array whose ops have no registry id. */ +const ScrArrOps *scr_arr_ops_overflow(const ScrArr *a); + +/* The element ops of `a`: NULL for every kind but SCR_ELEM_REF. */ +static inline const ScrArrOps *scr_arr_ops(const ScrArr *a) { + uint32_t id = (uint32_t)a->spare >> SCR_ARR_OPS_SHIFT; + if (__builtin_expect(id == SCR_ARR_OPS_OVERFLOW, 0)) return scr_arr_ops_overflow(a); + return scr_arr_ops_table[id]; +} /* Bytes per dense value slot. */ static inline size_t scr_arr_slot_width(const ScrArr *a) { diff --git a/packages/runtime/src/scr_async_dyn.c b/packages/runtime/src/scr_async_dyn.c index ec4dad794..a6cd813de 100644 --- a/packages/runtime/src/scr_async_dyn.c +++ b/packages/runtime/src/scr_async_dyn.c @@ -1274,7 +1274,7 @@ static void scr_dyn_all_result(ScrPromise *destination, ScrPromise *source) { scr_promise_fulfill_ref(destination, result, scr_dyn_retain_v, scr_dyn_release_v, scr_dyn_trace_v); } -static const ScrArrOps scr_dyn_arr_ops = { scr_dyn_retain_v, scr_dyn_release_v, scr_dyn_trace_v }; +static ScrArrOps scr_dyn_arr_ops = { scr_dyn_retain_v, scr_dyn_release_v, scr_dyn_trace_v }; /* ── Thenable adoption (PromiseResolve over a checked-dynamic value) ── * A non-promise object whose `then` is callable resolves through diff --git a/packages/runtime/src/scr_bytes_io.c b/packages/runtime/src/scr_bytes_io.c index eb3661fc6..5532d96da 100644 --- a/packages/runtime/src/scr_bytes_io.c +++ b/packages/runtime/src/scr_bytes_io.c @@ -6,8 +6,8 @@ * exact source lists). */ #include "scr_runtime.h" -static const ScrArrOps scr_bytes_io_str_ops = { scr_str_retain_v, scr_str_release_v, NULL }; -static const ScrArrOps scr_bytes_io_bytes_ops = { scr_bytes_retain_v, scr_bytes_release_v, NULL }; +static ScrArrOps scr_bytes_io_str_ops = { scr_str_retain_v, scr_str_release_v, NULL }; +static ScrArrOps scr_bytes_io_bytes_ops = { scr_bytes_retain_v, scr_bytes_release_v, NULL }; #include #include diff --git a/packages/runtime/src/scr_copying.c b/packages/runtime/src/scr_copying.c index 6864a6262..df4da1738 100644 --- a/packages/runtime/src/scr_copying.c +++ b/packages/runtime/src/scr_copying.c @@ -23,7 +23,7 @@ static bool copying_elem_is_ref(ScrElemKind kind) { static ScrArr *copying_arr_new_like(const ScrArr *a, size_t cap) { ScrArr *out = a->elem == SCR_ELEM_REF - ? scr_arr_new_ops(a->ops, cap ? cap : 1) + ? scr_arr_new_ops(scr_arr_ops(a), cap ? cap : 1) : scr_arr_new(a->elem, cap ? cap : 1); out->len = cap; return out; @@ -35,7 +35,7 @@ static uint64_t copying_arr_retain_slot(const ScrArr *a, uint64_t slot) { if (a->elem == SCR_ELEM_STR) p = scr_str_retain((ScrStr *)p); else if (a->elem == SCR_ELEM_ARR) p = scr_arr_retain((ScrArr *)p); else if (a->elem == SCR_ELEM_BYTES) p = scr_bytes_retain((ScrBytes *)p); - else p = a->ops->retain(p); + else p = scr_arr_ops(a)->retain(p); return copying_slot_from_ptr(p); } diff --git a/packages/runtime/src/scr_cycle.c b/packages/runtime/src/scr_cycle.c index ce4dfa5fa..682f8bfcd 100644 --- a/packages/runtime/src/scr_cycle.c +++ b/packages/runtime/src/scr_cycle.c @@ -1239,7 +1239,7 @@ void scr_cyc_on_release(void *obj) { * cycle-capable array and through the type's trace otherwise. */ #define SCR_CYC_CHILDREN(obj, type, ARRAY_WALK, VISIT, depth) \ do { \ - if ((type)->kind == SCR_CYC_KIND_ARRAY && ((const ScrArr *)(obj))->ext == NULL) \ + if ((type)->kind == SCR_CYC_KIND_ARRAY && !(((const ScrArr *)(obj))->spare & SCR_ARR_HAS_EXT)) \ ARRAY_WALK((const ScrArr *)(obj), (void *)(depth)); \ else \ (type)->trace((obj), (VISIT), (void *)(depth)); \ diff --git a/packages/runtime/src/scr_events_emitter.c b/packages/runtime/src/scr_events_emitter.c index c11b7897d..901e2d114 100644 --- a/packages/runtime/src/scr_events_emitter.c +++ b/packages/runtime/src/scr_events_emitter.c @@ -1352,7 +1352,7 @@ ScrArr *scr_emitter_event_names(ScrEmitter *em) { * The once wrapper is runtime-internal here, so BOTH forms answer the * original listeners (SEMANTICS.md documents the rawListeners * divergence). */ -static const ScrArrOps scr_ee_closure_arr_ops = { &scr_closure_retain_v, &scr_closure_release_v, +static ScrArrOps scr_ee_closure_arr_ops = { &scr_closure_retain_v, &scr_closure_release_v, &scr_closure_trace_v }; ScrArr *scr_emitter_listeners(ScrEmitter *em, ScrStr *name) { diff --git a/packages/runtime/src/scr_itable.c b/packages/runtime/src/scr_itable.c index 7b5202b81..a333aab95 100644 --- a/packages/runtime/src/scr_itable.c +++ b/packages/runtime/src/scr_itable.c @@ -474,11 +474,11 @@ static ScrArrState scr_itable_element(ScrITableElements *w, size_t i, uint64_t * *slot = a->data[i]; return (ScrArrState)*scr_arr_state_ptr(a, i); } - if (a->ext) { - while (w->sparse < a->ext->sparse_len && a->ext->sparse[w->sparse].index < i) w->sparse++; - if (w->sparse < a->ext->sparse_len && a->ext->sparse[w->sparse].index == i) { - *slot = a->ext->sparse[w->sparse].slot; - return (ScrArrState)a->ext->sparse[w->sparse].state; + if (scr_arr_ext_of(a)) { + while (w->sparse < scr_arr_ext_of(a)->sparse_len && scr_arr_ext_of(a)->sparse[w->sparse].index < i) w->sparse++; + if (w->sparse < scr_arr_ext_of(a)->sparse_len && scr_arr_ext_of(a)->sparse[w->sparse].index == i) { + *slot = scr_arr_ext_of(a)->sparse[w->sparse].slot; + return (ScrArrState)scr_arr_ext_of(a)->sparse[w->sparse].state; } } return SCR_ARR_HOLE; diff --git a/packages/runtime/src/scr_json.c b/packages/runtime/src/scr_json.c index d18ce85e4..adcc96560 100644 --- a/packages/runtime/src/scr_json.c +++ b/packages/runtime/src/scr_json.c @@ -12445,7 +12445,7 @@ ScrDynEntryPairs *scr_dyn_obj_entry_pairs(const ScrDyn *v) { void scr_dyn_entry_pairs_free(ScrDynEntryPairs *pairs) { free(pairs); } -static const ScrArrOps scr_dyn_value_list_ops = {scr_dyn_retain_v, scr_dyn_release_v, scr_dyn_trace_v}; +static ScrArrOps scr_dyn_value_list_ops = {scr_dyn_retain_v, scr_dyn_release_v, scr_dyn_trace_v}; /* A native array of `count` owned dyn values, moved in with one dense push. * Frees `values` unless it is `stack`. */ diff --git a/packages/runtime/src/scr_json.test.c b/packages/runtime/src/scr_json.test.c index 3946331be..736eca540 100644 --- a/packages/runtime/src/scr_json.test.c +++ b/packages/runtime/src/scr_json.test.c @@ -520,7 +520,7 @@ static void test_item_release(void *ptr) { static ScrArr *test_tags_new(size_t cap) { return scr_arr_new(SCR_ELEM_STR, cap); } static ScrArr *test_list_new(size_t cap) { - static const ScrArrOps item_ops = { test_item_retain, test_item_release, NULL }; + static ScrArrOps item_ops = { test_item_retain, test_item_release, NULL }; return scr_arr_new_ops(&item_ops, cap); } static void test_arr_release(void *ptr) { scr_arr_release(ptr); } diff --git a/packages/runtime/src/scr_lib.c b/packages/runtime/src/scr_lib.c index db0926690..3441e6142 100644 --- a/packages/runtime/src/scr_lib.c +++ b/packages/runtime/src/scr_lib.c @@ -5788,7 +5788,7 @@ void scr_fs_symlink(ScrStr *target, ScrStr *destination, int kind) { if (!scr_fs_path_ok(target, "target")) return; if (!scr_fs_path_ok(destination, "path")) return; #ifdef _WIN32 - static const ScrArrOps str_ops = { scr_str_retain_v, scr_str_release_v, NULL }; + static ScrArrOps str_ops = { scr_str_retain_v, scr_str_release_v, NULL }; ScrArr *parts = scr_arr_new_ops(&str_ops, 3); scr_arr_push_ref(parts, scr_str_retain(destination)); scr_arr_push_ref(parts, scr_str_new("..", 2)); diff --git a/packages/runtime/src/scr_map.c b/packages/runtime/src/scr_map.c index 5d4935ed0..99caa31a8 100644 --- a/packages/runtime/src/scr_map.c +++ b/packages/runtime/src/scr_map.c @@ -1558,7 +1558,7 @@ void scr_map_values_into(ScrMap *set, const ScrMap *source, bool checked) { } ScrArr *scr_set_to_arr_dyn(const ScrMap *map) { - static const ScrArrOps dyn_ops = { scr_dyn_retain_v, scr_dyn_release_v, scr_dyn_trace_v }; + static ScrArrOps dyn_ops = { scr_dyn_retain_v, scr_dyn_release_v, scr_dyn_trace_v }; ScrArr *out = scr_arr_new_ops(&dyn_ops, map->nlive); for (size_t i = 0; i < map->nentries; i++) { if (scr_map_iter_live(map, (double)i)) scr_arr_push_ref(out, scr_map_dyn_key(map, (double)i)); diff --git a/packages/runtime/src/scr_publish.c b/packages/runtime/src/scr_publish.c index 025e13a51..675a13ca9 100644 --- a/packages/runtime/src/scr_publish.c +++ b/packages/runtime/src/scr_publish.c @@ -201,7 +201,7 @@ static void scr_pub_slot(ScrPub *ctx, uint64_t slot, ScrPubFn fn) { void scr_pub_arr(void *v, ScrPub *ctx, ScrPubFn elem) { ScrArr *a = v; - if (a->ext && (a->ext->props || a->ext->metadata)) { + if (scr_arr_ext_of(a) && (scr_arr_ext_of(a)->props || scr_arr_ext_of(a)->metadata)) { scr_pub_refuse(ctx, "Cannot publish an array with non-index properties"); return; } @@ -212,9 +212,9 @@ void scr_pub_arr(void *v, ScrPub *ctx, ScrPubFn elem) { const uint8_t *states = scr_arr_states_end(a); for (size_t i = 0; i < dense; i++) if (!states || *scr_arr_state_in(states, i)) scr_pub_slot(ctx, a->data[i], fn); - if (a->ext) - for (size_t i = 0; i < a->ext->sparse_len; i++) - if (a->ext->sparse[i].state) scr_pub_slot(ctx, a->ext->sparse[i].slot, fn); + if (scr_arr_ext_of(a)) + for (size_t i = 0; i < scr_arr_ext_of(a)->sparse_len; i++) + if (scr_arr_ext_of(a)->sparse[i].state) scr_pub_slot(ctx, scr_arr_ext_of(a)->sparse[i].slot, fn); } /* publish() of a scalar Node.js refuses (a Date): nothing to walk. */ diff --git a/packages/runtime/src/scr_regex.c b/packages/runtime/src/scr_regex.c index 73440c110..b62cba896 100644 --- a/packages/runtime/src/scr_regex.c +++ b/packages/runtime/src/scr_regex.c @@ -667,7 +667,7 @@ static void scr_regex_defer_metadata(ScrArr *row, ScrDyn *source, int index) { /* The pending match source of a result, or NULL once its metadata exists. */ static ScrRegexMatchSource *scr_regex_pending(const ScrArr *a) { - const ScrDyn *metadata = a->ext ? a->ext->metadata : NULL; + const ScrDyn *metadata = scr_arr_ext_of(a) ? scr_arr_ext_of(a)->metadata : NULL; if (!metadata || metadata->kind != SCR_DYN_HANDLE || metadata->v.handle.tag != SCR_DYNH_MATCH_SOURCE) return NULL; return metadata->v.handle.ptr; @@ -707,7 +707,7 @@ static ScrDyn *scr_regex_groups_of(const ScrArr *a, uint8_t *bc) { static void scr_regex_materialize_metadata(ScrArr *a) { ScrRegexMatchSource *src = scr_regex_pending(a); if (!src) return; - ScrArrExt *x = a->ext; + ScrArrExt *x = scr_arr_ext_of(a); ScrDyn *source = x->metadata; ScrDyn *props = scr_dyn_new_obj(); scr_dyn_obj_set(props, "index", 5, scr_dyn_new_num(x->match_index)); @@ -724,7 +724,7 @@ static bool scr_regex_copy_pending_metadata(ScrArr *a, ScrDyn *target) { scr_regex_materialize_metadata(a); return false; } - scr_dyn_arr_named_put(target, "index", 5, scr_dyn_new_num(a->ext->match_index)); + scr_dyn_arr_named_put(target, "index", 5, scr_dyn_new_num(scr_arr_ext_of(a)->match_index)); scr_dyn_arr_named_put(target, "input", 5, scr_regex_source_input(src)); scr_dyn_arr_named_put(target, "groups", 6, scr_regex_groups_of(a, scr_regex_bc(src->re))); return true; @@ -734,11 +734,11 @@ ScrDyn *scr_regex_result_prop(ScrArr *a, ScrStr *key) { ScrRegexMatchSource *src = scr_regex_pending(a); if (src) { if (key->len == 5 && memcmp(key->data, "index", 5) == 0) - return scr_dyn_new_num(a->ext->match_index); + return scr_dyn_new_num(scr_arr_ext_of(a)->match_index); if (key->len == 5 && memcmp(key->data, "input", 5) == 0) return scr_regex_source_input(src); scr_regex_materialize_metadata(a); } - ScrDyn *metadata = a->ext ? a->ext->metadata : NULL; + ScrDyn *metadata = scr_arr_ext_of(a) ? scr_arr_ext_of(a)->metadata : NULL; ScrDyn *value = metadata ? scr_dyn_obj_get(metadata, key->data, key->len) : NULL; return scr_dyn_retain(value ? value : scr_dyn_undefined()); } diff --git a/packages/runtime/src/scr_region.c b/packages/runtime/src/scr_region.c index 3a7dad541..dbd74f3ac 100644 --- a/packages/runtime/src/scr_region.c +++ b/packages/runtime/src/scr_region.c @@ -803,17 +803,17 @@ static void *scr_rgn_slot_ptr(uint64_t slot) { static void scr_rgn_walk_arr(ScrRgnWalk *w, ScrArr *a) { if (a->elem != SCR_ELEM_REF && a->elem != SCR_ELEM_ARR) return; ScrRgnSlotKind k = a->elem == SCR_ELEM_ARR ? (ScrRgnSlotKind){SCR_RGN_W_ARR, NULL} - : scr_rgn_slot_kind(a->ops ? a->ops->release : NULL); + : scr_rgn_slot_kind(scr_arr_ops(a) ? scr_arr_ops(a)->release : NULL); size_t dense = a->len < a->cap ? a->len : a->cap; const uint8_t *states = scr_arr_states_end(a); for (size_t i = 0; i < dense; i++) if (!states || *scr_arr_state_in(states, i) == SCR_ARR_VALUE) scr_rgn_slot_as(w, scr_rgn_slot_ptr(a->data[i]), k); - if (!a->ext) return; - for (size_t i = 0; i < a->ext->sparse_len; i++) - if (a->ext->sparse[i].state == SCR_ARR_VALUE) scr_rgn_slot_as(w, scr_rgn_slot_ptr(a->ext->sparse[i].slot), k); - for (size_t i = 0; i < a->ext->prop_len; i++) - if (a->ext->props[i].state == SCR_ARR_VALUE) scr_rgn_slot_as(w, scr_rgn_slot_ptr(a->ext->props[i].slot), k); - scr_rgn_enter(w, a->ext->metadata, SCR_RGN_W_DYN, NULL); + if (!scr_arr_ext_of(a)) return; + for (size_t i = 0; i < scr_arr_ext_of(a)->sparse_len; i++) + if (scr_arr_ext_of(a)->sparse[i].state == SCR_ARR_VALUE) scr_rgn_slot_as(w, scr_rgn_slot_ptr(scr_arr_ext_of(a)->sparse[i].slot), k); + for (size_t i = 0; i < scr_arr_ext_of(a)->prop_len; i++) + if (scr_arr_ext_of(a)->props[i].state == SCR_ARR_VALUE) scr_rgn_slot_as(w, scr_rgn_slot_ptr(scr_arr_ext_of(a)->props[i].slot), k); + scr_rgn_enter(w, scr_arr_ext_of(a)->metadata, SCR_RGN_W_DYN, NULL); } static void scr_rgn_walk_map(ScrRgnWalk *w, ScrMap *m) { diff --git a/packages/runtime/src/scr_region.test.c b/packages/runtime/src/scr_region.test.c index 31e98942a..ad1f92518 100644 --- a/packages/runtime/src/scr_region.test.c +++ b/packages/runtime/src/scr_region.test.c @@ -61,7 +61,7 @@ static void node_release(void *o) { if (c) c->count--; } -static const ScrArrOps node_ops = {node_retain, node_release, NULL}; +static ScrArrOps node_ops = {node_retain, node_release, NULL}; /* ── the "emitted" ordinary class (acyclic, plain heap block) ────────── */ diff --git a/packages/runtime/src/scr_stream.c b/packages/runtime/src/scr_stream.c index 7d7ec04ed..f7330b615 100644 --- a/packages/runtime/src/scr_stream.c +++ b/packages/runtime/src/scr_stream.c @@ -4420,7 +4420,7 @@ static ScrPromise *scr_sc_consume(ScrStream *s, int kind) { scr_promise_reject_pending(p); return p; } - static const ScrArrOps bytes_ops = { scr_bytes_retain_v, scr_bytes_release_v, NULL }; + static ScrArrOps bytes_ops = { scr_bytes_retain_v, scr_bytes_release_v, NULL }; ScrArr *chunks = scr_arr_new_ops(&bytes_ops, 4); ScrStreamErrInv fin_inv = kind == SCR_SC_TEXT ? &scr_sc_fin_text : kind == SCR_SC_JSON ? &scr_sc_fin_json diff --git a/packages/runtime/test/test_array.c b/packages/runtime/test/test_array.c index 3eea84dcf..02da2fac2 100644 --- a/packages/runtime/test/test_array.c +++ b/packages/runtime/test/test_array.c @@ -19,7 +19,7 @@ #include /* Sparse entries live in the array's out-of-line storage. */ -static size_t arr_sparse_len(const ScrArr *a) { return a->ext ? a->ext->sparse_len : 0; } +static size_t arr_sparse_len(const ScrArr *a) { return scr_arr_ext_of(a) ? scr_arr_ext_of(a)->sparse_len : 0; } #ifdef SCR_RC_AUDIT long scr_str_live_count(void); /* provided by scr_string.c */ @@ -396,8 +396,8 @@ static MockRec *mock_cyc_new(int value) { return r; } -static const ScrArrOps mock_ops = { &mock_retain, &mock_release, NULL }; -static const ScrArrOps mock_cyc_ops = { &mock_cyc_retain, &mock_cyc_release, &mock_trace }; +static ScrArrOps mock_ops = { &mock_retain, &mock_release, NULL }; +static ScrArrOps mock_cyc_ops = { &mock_cyc_retain, &mock_cyc_release, &mock_trace }; static void test_ref_elements(void) { #ifdef SCR_RC_AUDIT @@ -444,6 +444,58 @@ static void test_ref_elements(void) { #endif } +/* Element ops past the registry's capacity: every descriptor still finds its + * own retain and release, and the first ones keep their registry ids. */ +static void test_ops_overflow(void) { + enum { N = SCR_ARR_OPS_OVERFLOW + 8 }; + static ScrArrOps many[N]; + static ScrArr *arrays[N]; + for (int i = 0; i < N; i++) many[i] = (ScrArrOps){ &mock_retain, &mock_release, NULL, 0 }; + for (int i = 0; i < N; i++) { + arrays[i] = scr_arr_new_ops(&many[i], 0); + scr_arr_push_ref(arrays[i], mock_new(i)); + } + check(scr_arr_ops(arrays[0]) == &many[0], "a registered descriptor answers by id"); + check(scr_arr_ops(arrays[N - 1]) == &many[N - 1], "an overflow descriptor answers through the array"); + check(many[N - 1].id == SCR_ARR_OPS_OVERFLOW, "descriptors past the table get the overflow id"); + MockRec *r = (MockRec *)scr_arr_get_ref(arrays[N - 1], 0); + check(r->rc == 2 && r->value == N - 1, "an overflow array retains through its own ops"); + mock_release(r); + for (int i = 0; i < N; i++) scr_arr_release(arrays[i]); + check(mock_live == 0, "overflow arrays release their elements"); +} + +/* An array with out-of-line storage keeps it through every change of its + * dense allocation: growth, head room, unshift into new room, sorting + * (which rebuilds the storage) and a sparse tail. */ +static void test_ext_through_storage_changes(void) { + ScrArr *a = scr_arr_new(SCR_ELEM_F64, 0); + scr_arr_set_f64(a, -1, 7); /* a noncanonical property: out-of-line storage */ + check(scr_arr_ext_of(a) != NULL, "a numeric property creates out-of-line storage"); + for (int i = 0; i < 100; i++) scr_arr_push_f64(a, 100 - i); + check_f64(scr_arr_get_f64(a, -1), 7, "the property survives growth"); + for (int i = 0; i < 40; i++) scr_arr_shift_f64(a); + check_f64(scr_arr_get_f64(a, -1), 7, "the property survives head room"); + for (int i = 0; i < 60; i++) scr_arr_unshift_f64(a, 1000 + i); + check_f64(scr_arr_len(a), 120, "unshift after shifts keeps the length"); + check_f64(scr_arr_get_f64(a, -1), 7, "the property survives new head room"); + ScrArr *same = scr_arr_sort_primitive(a, false); + check(same == a, "an in-place sort returns the array"); + check_f64(scr_arr_get_f64(a, 0), 1, "sorting reorders the values"); + check_f64(scr_arr_get_f64(a, -1), 7, "the property survives a sort"); + scr_arr_set_f64(a, 5000000, 9); /* a sparse tail entry */ + check(arr_sparse_len(a) == 1 && scr_arr_get_f64(a, 5000000) == 9, "a sparse entry joins the out-of-line storage"); + scr_arr_release(same); /* the sort's result reference */ + scr_arr_release(a); + + ScrArr *empty = scr_arr_new(SCR_ELEM_F64, 0); + scr_arr_set_f64(empty, 0.5, 3); + check_f64(scr_arr_get_f64(empty, 0.5), 3, "an array without elements keeps a property"); + scr_arr_push_f64(empty, 1); + check_f64(scr_arr_get_f64(empty, 0.5), 3, "the property survives the first element"); + scr_arr_release(empty); +} + static void test_ref_cycle(void) { #ifdef SCR_RC_AUDIT long arrays0 = scr_arr_live_count(); @@ -1522,7 +1574,11 @@ static void test_inline_slots(void) { scr_arr_push_ref(words, scr_str_new("a", 1)); scr_arr_push_ref(words, scr_str_new("b", 1)); scr_arr_set_ref(words, 3000000, scr_str_new("far", 3)); - check(inline_slots(words) && arr_sparse_len(words) == 1, "a far index leaves inline slots in place"); + /* A sparse entry needs out-of-line storage, whose pointer lives before the + * dense allocation: the inline slots move out of line with their values. */ + check(!inline_slots(words) && arr_sparse_len(words) == 1 && + strcmp(((ScrStr *)scr_arr_peek_ref(words, 1))->data, "b") == 0, + "a far index moves the inline slots out of line"); scr_arr_unshift_ref(words, scr_str_new("z", 1)); check(!inline_slots(words) && scr_arr_len(words) == 3000002 && strcmp(((ScrStr *)scr_arr_peek_ref(words, 0))->data, "z") == 0 && @@ -2149,6 +2205,8 @@ int main(int argc, char **argv) { test_nested_rc(); test_index_of_includes(); test_ref_elements(); + test_ops_overflow(); + test_ext_through_storage_changes(); test_ref_cycle(); test_ref_truncate_cycle(); test_ref_trace_storage_boundaries(); diff --git a/packages/runtime/test/test_lib.c b/packages/runtime/test/test_lib.c index db8d29693..6bdbb085d 100644 --- a/packages/runtime/test/test_lib.c +++ b/packages/runtime/test/test_lib.c @@ -22,7 +22,7 @@ #include /* Sparse entries live in the array's out-of-line storage. */ -static size_t arr_sparse_len(const ScrArr *a) { return a->ext ? a->ext->sparse_len : 0; } +static size_t arr_sparse_len(const ScrArr *a) { return scr_arr_ext_of(a) ? scr_arr_ext_of(a)->sparse_len : 0; } #endif static int checks = 0; diff --git a/packages/runtime/test/test_map.c b/packages/runtime/test/test_map.c index 9957a994c..32bdb7be1 100644 --- a/packages/runtime/test/test_map.c +++ b/packages/runtime/test/test_map.c @@ -636,7 +636,7 @@ static void test_value_keys_and_copies(void) { } #ifdef SCR_RC_AUDIT -static const ScrArrOps map_arr_ops = { scr_map_retain_v, scr_map_release_v, scr_map_trace_v }; +static ScrArrOps map_arr_ops = { scr_map_retain_v, scr_map_release_v, scr_map_trace_v }; static void test_identity_cycles(void) { scr_collect_cycles(); diff --git a/packages/runtime/test/test_regex.c b/packages/runtime/test/test_regex.c index 6084c5102..6afa8391e 100644 --- a/packages/runtime/test/test_regex.c +++ b/packages/runtime/test/test_regex.c @@ -333,7 +333,7 @@ static void test_exec_groups(void) { check(full && lean && full->len == lean->len && lean->len == 2, "exec_groups keeps the capture count"); check(*scr_arr_state_ptr(lean, 0) == SCR_ARR_UNDEFINED && *scr_arr_state_ptr(lean, 1) == SCR_ARR_UNDEFINED, "exec_groups: unselected and nonparticipating captures are undefined"); - check(lean->ext == NULL || lean->ext->metadata == NULL, "exec_groups: no index/input properties"); + check(scr_arr_ext_of(lean) == NULL || scr_arr_ext_of(lean)->metadata == NULL, "exec_groups: no index/input properties"); scr_arr_release(full); scr_arr_release(lean); ScrStr *ab = S("ab"); From 5c851762d6d96b263557c61d79d88c595916a674 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 13:12:24 -0500 Subject: [PATCH 43/51] Read ASCII characters of strings inline in charAt and indexing charAt and s[i] on a string proven ASCII at an exact in-range index now return the character from a module-level table of the 128 one-character strings, emitted with the helper as immortal literals with the same capacity word and key hash as the runtime's single-character statics. An exact integer index (a counted-loop counter) skips the double conversion. Every other string or index calls scr_str_char_at as before. --- .../src/backend/llvm/expr-containers.ts | 25 ++++-- .../src/backend/llvm/string-lifetimes.test.ts | 3 +- .../compiler/src/backend/llvm/string-reads.ts | 86 +++++++++++++++++++ .../test/string-char-at-emission.test.ts | 55 ++++++++++++ 4 files changed, 160 insertions(+), 9 deletions(-) create mode 100644 packages/compiler/test/string-char-at-emission.test.ts diff --git a/packages/compiler/src/backend/llvm/expr-containers.ts b/packages/compiler/src/backend/llvm/expr-containers.ts index bf8d502a1..8d1d1d28f 100644 --- a/packages/compiler/src/backend/llvm/expr-containers.ts +++ b/packages/compiler/src/backend/llvm/expr-containers.ts @@ -45,6 +45,8 @@ import { emitBorrowedInput } from "./borrowed-inputs.js"; import { emitFusedMapSet, rememberMapEntry } from "./map-entry-fusion.js"; import { emitScratchStringSlice, emitStringSliceRead } from "./string-slices.js"; import { + STRING_CHAR_AT_DECL, + STRING_CHAR_AT_INT_MARKER, STRING_CHAR_CODE_AT_DECL, STRING_CHAR_CODE_AT_INT_MARKER, STRING_UTF16_LEN_DECL, @@ -325,14 +327,21 @@ export function emitStrIntrinsic( B.line(`${t} = call double @sc_str_char_code_at(ptr ${r.name}, double ${args[0]!.name})`); return { name: t, type: e.type }; } - case "charAt": - return call( - "scr_str_char_at", - "ptr (ptr, double)", - `ptr ${r.name}, double ${args[0]!.name}`, - "ptr", - true, - ); + case "charAt": { + // sc_str_char_at (string-reads.ts) answers proven-ASCII strings inline. + host.declare(STRING_CHAR_AT_DECL); + const t = B.tmp(); + const index = e.args[0] ? exactInteger(host, args[0]!, e.args[0]) : null; + if (index) { + host.declare(STRING_CHAR_AT_INT_MARKER); + const wide = widenInteger(host, index); + B.line(`${t} = call ptr @sc_str_char_at_int(ptr ${r.name}, i64 ${wide})`); + } else { + const at = args[0]?.name ?? f64Lit(0); + B.line(`${t} = call ptr @sc_str_char_at(ptr ${r.name}, double ${at})`); + } + return host.own({ name: t, type: e.type }); + } case "indexOf": return call( "scr_str_index_of", diff --git a/packages/compiler/src/backend/llvm/string-lifetimes.test.ts b/packages/compiler/src/backend/llvm/string-lifetimes.test.ts index 8622085b4..f85addcf8 100644 --- a/packages/compiler/src/backend/llvm/string-lifetimes.test.ts +++ b/packages/compiler/src/backend/llvm/string-lifetimes.test.ts @@ -311,7 +311,8 @@ const methodCases: { // enters through the double ABI. { method: "charCodeAt", args: [num(1)], result: F64, target: "char_code_at_int", inline: true }, { method: "charCodeAt", args: [num(1.5)], result: F64, target: "char_code_at", inline: true }, - { method: "charAt", args: [num(1)], result: STRING, target: "char_at" }, + { method: "charAt", args: [num(1)], result: STRING, target: "char_at_int", inline: true }, + { method: "charAt", args: [num(1.5)], result: STRING, target: "char_at", inline: true }, { method: "indexOf", args: [ref("argument"), num(1)], result: F64, target: "index_of" }, { method: "includes", args: [ref("argument")], result: BOOL, target: "includes" }, { diff --git a/packages/compiler/src/backend/llvm/string-reads.ts b/packages/compiler/src/backend/llvm/string-reads.ts index d7bbed272..03a9fa282 100644 --- a/packages/compiler/src/backend/llvm/string-reads.ts +++ b/packages/compiler/src/backend/llvm/string-reads.ts @@ -18,6 +18,8 @@ */ import { exactIndexLines, sizeToDouble } from "./common.js"; +import { wtf8Bytes } from "../literals.js"; +import { literalCapWord, literalCapWord32 } from "./string-key-hash.js"; export const STRING_UTF16_LEN_DECL = "declare double @scr_str_utf16_len(ptr)"; export const STRING_CHAR_CODE_AT_DECL = "declare double @scr_str_char_code_at(ptr, double)"; @@ -31,6 +33,9 @@ const READ_CALL_ATTRS = "cold nocallback nounwind willreturn"; /** Requests the integer-index helper below; an LLVM comment line among the * declarations, so it never declares anything itself. */ export const STRING_CHAR_CODE_AT_INT_MARKER = "; uses @sc_str_char_code_at_int"; +export const STRING_CHAR_AT_DECL = "declare ptr @scr_str_char_at(ptr, double)"; +/** Requests the integer-index charAt helper, as the marker above does. */ +export const STRING_CHAR_AT_INT_MARKER = "; uses @sc_str_char_at_int"; /** Load the proven-ASCII flag of `%s` into `%ascii`. */ function asciiTest(): string[] { @@ -85,6 +90,34 @@ function scanRead(): string[] { ]; } +/** The one-character strings of the ASCII range as immortal literals + * ({ rc = SIZE_MAX, len = 1, capacity word, char, NUL }), indexed by + * character code: the inline charAt result. Their bytes, capacity words + * and key hashes equal those of the runtime's single-character statics, so + * they are interchangeable with the strings the runtime returns. */ +function asciiCharTable(sz: string): string[] { + const type = `{ ${sz}, ${sz}, ${sz}, [2 x i8] }`; + const entries: string[] = []; + for (let c = 0; c < 128; c++) { + const bytes = wtf8Bytes(String.fromCharCode(c)); + const word = sz === "i64" ? literalCapWord(bytes) : literalCapWord32(bytes); + entries.push(`${type} { ${sz} -1, ${sz} 1, ${sz} ${word}, [2 x i8] [i8 ${c}, i8 0] }`); + } + return [`@sc_ascii_chars = internal constant [128 x ${type}] [${entries.join(", ")}]`]; +} + +/** The table entry for byte `%byte` of a proven-ASCII string. */ +function asciiCharRead(sz: string): string[] { + return [ + ` %data = getelementptr inbounds %ScrStr, ptr %s, i32 1`, + ` %p = getelementptr inbounds i8, ptr %data, i64 %k`, + ` %byte = load i8, ptr %p`, + ` %code = zext i8 %byte to i64`, + ` %char = getelementptr inbounds [128 x { ${sz}, ${sz}, ${sz}, [2 x i8] }], ptr @sc_ascii_chars, i64 0, i64 %code`, + ` ret ptr %char`, + ]; +} + export interface StringReadHelper { decl: string; define(sz: string, attrs: string, scan: boolean): string[]; @@ -200,4 +233,57 @@ export const STRING_READ_HELPERS: readonly StringReadHelper[] = [ `}`, ], }, + { + // charAt of a proven-ASCII string at an exact in-range index is the + // table entry for its byte; every other case calls the runtime, which + // keeps the complete semantics. The table is defined with the helper. + decl: STRING_CHAR_AT_DECL, + define: (sz, attrs) => { + let n = 0; + const exactIndex = exactIndexLines("%i", "%len", sz, () => `%ix${n++}`); + return [ + ...asciiCharTable(sz), + ``, + `define internal ptr @sc_str_char_at(ptr %s, double %i) ${attrs} {`, + `entry:`, + ...asciiTest(), + ` br i1 %ascii, label %bounds, label %slow`, + `bounds:`, + ...loadLength(sz), + ...exactIndex.lines.map((line) => ` ${line}`), + ` br i1 ${exactIndex.ok}, label %read, label %slow`, + `read:`, + ` %k = add i64 ${exactIndex.wide}, 0`, + ...asciiCharRead(sz), + `slow:`, + ` %r = call ptr @scr_str_char_at(ptr %s, double %i) ${READ_CALL_ATTRS}`, + ` ret ptr %r`, + `}`, + ]; + }, + }, + { + // An exact integer index, as for sc_str_char_code_at_int. Requested + // together with STRING_CHAR_AT_DECL, which defines the table. + decl: STRING_CHAR_AT_INT_MARKER, + define: (sz, attrs) => [ + `define internal ptr @sc_str_char_at_int(ptr %s, i64 %i) ${attrs} {`, + `entry:`, + ...asciiTest(), + ` br i1 %ascii, label %bounds, label %slow`, + `bounds:`, + ...loadLength(sz), + ...(sz === "i64" ? [] : [` %len64 = zext ${sz} %len to i64`]), + ` %ok = icmp ult i64 %i, ${sz === "i64" ? "%len" : "%len64"}`, + ` br i1 %ok, label %read, label %slow`, + `read:`, + ` %k = add i64 %i, 0`, + ...asciiCharRead(sz), + `slow:`, + ` %d = sitofp i64 %i to double`, + ` %r = call ptr @scr_str_char_at(ptr %s, double %d) ${READ_CALL_ATTRS}`, + ` ret ptr %r`, + `}`, + ], + }, ]; diff --git a/packages/compiler/test/string-char-at-emission.test.ts b/packages/compiler/test/string-char-at-emission.test.ts new file mode 100644 index 000000000..8e5814120 --- /dev/null +++ b/packages/compiler/test/string-char-at-emission.test.ts @@ -0,0 +1,55 @@ +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { expect, test } from "vitest"; +import { compile, deserializeModule, validateModule } from "../src/index.js"; +import { emitLlvmModule } from "../src/backend/llvm/emitter.js"; +import { literalCapWord } from "../src/backend/llvm/string-key-hash.js"; +import type { IrModule } from "../src/ir/ir.js"; + +async function lower(source: string): Promise { + const dir = await mkdtemp(join(tmpdir(), "scriptc-char-at-")); + try { + const entry = join(dir, "main.ts"); + const outPath = join(dir, "main.ir.json"); + await writeFile(entry, source); + const result = await compile(entry, { outDir: dir, outPath, outputKind: "ir" }); + if (!result.ok) + throw new Error(result.diagnostics.map((d) => `${d.code}: ${d.message}`).join("\n")); + const mod = deserializeModule(await readFile(outPath, "utf8")); + expect(validateModule(mod)).toEqual([]); + return mod; + } finally { + await rm(dir, { recursive: true, force: true }); + } +} + +const options = { targetTriple: "x86_64-unknown-linux-gnu", pointerBits: 64 } as const; + +test("charAt reads proven-ASCII strings through the module's character table", async () => { + const mod = await lower( + 'const s = process.argv.length > 9 ? "x" : "hello";\nconst i: number = process.argv.length - 1;\nconsole.log(s.charAt(i), s[i]);\n', + ); + const ir = emitLlvmModule(mod, options); + expect(ir).toContain("define internal ptr @sc_str_char_at(ptr %s, double %i)"); + expect(ir).toContain("call ptr @scr_str_char_at(ptr %s, double %i) cold"); + // Each entry is an immortal one-character literal with the runtime's + // capacity word (length, proven-ASCII bit and key hash). + const word = literalCapWord(new Uint8Array([65])); + expect(ir).toContain(`{ i64 -1, i64 1, i64 ${word}, [2 x i8] [i8 65, i8 0] }`); + expect(ir.match(/^@sc_ascii_chars = internal constant \[128 x /gm)).toHaveLength(1); +}); + +test("charAt at a counted-loop index uses the integer helper", async () => { + const mod = await lower( + 'const s = process.argv.length > 9 ? "x" : "hello";\nlet out = "";\nfor (let i = 0; i < s.length; i++) out += s.charAt(i);\nconsole.log(out);\n', + ); + const ir = emitLlvmModule(mod, options); + expect(ir).toContain("define internal ptr @sc_str_char_at_int(ptr %s, i64 %i)"); + expect(ir).toMatch(/call ptr @sc_str_char_at_int\(ptr %\w+, i64 %\w+\)/); +}); + +test("programs without charAt carry no character table", async () => { + const mod = await lower('console.log("hello".length + process.argv.length);\n'); + expect(emitLlvmModule(mod, options)).not.toContain("@sc_ascii_chars"); +}); From a4d32a8e38fd765dc688ff057110c8ace10b7d3a Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 13:22:56 -0500 Subject: [PATCH 44/51] Use the default TLS version bound for null minVersion and maxVersion in tls.createServer tls.createServer validated minVersion and maxVersion with its own loop, which threw ERR_TLS_INVALID_PROTOCOL_VERSION for null. It now uses the same check as tls.connect, where null and undefined select the default bound, as in Node. --- packages/runtime/src/scr_tls.c | 37 ++++-------------------- tests/corpus/6722-tls-min-max-version.ts | 18 +++++++++++- 2 files changed, 22 insertions(+), 33 deletions(-) diff --git a/packages/runtime/src/scr_tls.c b/packages/runtime/src/scr_tls.c index 59f3c2c0a..526dd7760 100644 --- a/packages/runtime/src/scr_tls.c +++ b/packages/runtime/src/scr_tls.c @@ -1813,38 +1813,11 @@ static bool scr_tls_opts_validate(const ScrDyn *opts) { return false; } } - static const char *const VERSION_OPTS[] = { "minVersion", "maxVersion", NULL }; - for (size_t i = 0; VERSION_OPTS[i] != NULL; i++) { - const ScrDyn *v = scr_dyn_obj_get(opts, VERSION_OPTS[i], strlen(VERSION_OPTS[i])); - if (v == NULL || v->kind == SCR_DYN_UNDEF) continue; - bool valid = false; - if (v->kind == SCR_DYN_STR) { - static const char *const KNOWN[] = { "TLSv1", "TLSv1.1", "TLSv1.2", "TLSv1.3", NULL }; - for (size_t k = 0; KNOWN[k] != NULL; k++) { - if (v->v.str->len == strlen(KNOWN[k]) && memcmp(v->v.str->data, KNOWN[k], v->v.str->len) == 0) { - valid = true; - break; - } - } - } - if (!valid) { - /* %j: strings render JSON-quoted, everything else inspect-lite. */ - char rendered[96]; - if (v->kind == SCR_DYN_STR) { - snprintf(rendered, sizeof rendered, "\"%.*s\"", - (int)(v->v.str->len < 80 ? v->v.str->len : 80), v->v.str->data); - } else { - /* %j over non-strings: JSON.stringify's scalar renderings. */ - char lite[64]; - snprintf(rendered, sizeof rendered, "%s", scr_dyn_inspect_lite(v, lite, sizeof lite)); - } - char msg[192]; - int len = snprintf(msg, sizeof msg, "%s is not a valid %s TLS protocol version", - rendered, VERSION_OPTS[i][2] == 'n' ? "minimum" : "maximum"); - scr_throw_error_msg_code(SCR_ERR_TYPE, msg, (size_t)len, "ERR_TLS_INVALID_PROTOCOL_VERSION"); - return false; - } - } + /* Node's toV: null and undefined select the default bound. */ + int version; + if (!scr_tls_version_dyn(scr_dyn_obj_get(opts, "minVersion", 10), "minimum", 12, &version) || + !scr_tls_version_dyn(scr_dyn_obj_get(opts, "maxVersion", 10), "maximum", 13, &version)) + return false; static const char *const NUM_OPTS[] = { "handshakeTimeout", "keepAliveInitialDelay", NULL }; for (size_t i = 0; NUM_OPTS[i] != NULL; i++) { const ScrDyn *v = scr_dyn_obj_get(opts, NUM_OPTS[i], strlen(NUM_OPTS[i])); diff --git a/tests/corpus/6722-tls-min-max-version.ts b/tests/corpus/6722-tls-min-max-version.ts index 07d01867d..70cddae52 100644 --- a/tests/corpus/6722-tls-min-max-version.ts +++ b/tests/corpus/6722-tls-min-max-version.ts @@ -4,7 +4,8 @@ // ERR_SSL_TLSV1_ALERT_PROTOCOL_VERSION, a client range with neither TLSv1.2 // nor TLSv1.3 fails with ERR_SSL_NO_PROTOCOLS_AVAILABLE once the handshake // starts, and an invalid version or a secureProtocol conflict throws Node's -// TypeError from tls.connect. +// TypeError from tls.connect and tls.createServer. null selects the default +// bound, as undefined does. import * as tls from "node:tls"; import { readFileSync } from "node:fs"; @@ -16,6 +17,17 @@ const key = readFileSync(dir + "localhost-key.pem"); const upTo12 = tls.createServer({ cert, key, maxVersion: "TLSv1.2" }, (s) => s.end(`server:${s.getProtocol()}`)); const from13 = tls.createServer({ cert, key, minVersion: "TLSv1.3" }, (s) => s.end(`server:${s.getProtocol()}`)); const any = tls.createServer({ cert, key }, (s) => s.end(`server:${s.getProtocol()}`)); +const nullBounds = tls.createServer({ cert, key, minVersion: null as any, maxVersion: null as any }, (s) => + s.end(`server:${s.getProtocol()}`), +); +for (const maxVersion of [3, "TLSv9"]) { + try { + tls.createServer({ cert, key, maxVersion: maxVersion as any }); + } catch (e) { + const err = e as NodeJS.ErrnoException; + console.log("createServer", err.name, err.code, err.message); + } +} function listen(server: tls.Server): Promise { return new Promise((resolve) => @@ -74,12 +86,16 @@ async function main(): Promise { await attempt("max-1.1", pAny, { maxVersion: "TLSv1.1" }); await attempt("min-above-max", pAny, { minVersion: "TLSv1.3", maxVersion: "TLSv1.2" }); await attempt("null-min", pAny, { minVersion: null }); + const pNull = await listen(nullBounds); + await attempt("null-server", pNull, {}); + await attempt("null-both", pNull, { minVersion: null, maxVersion: null }); await attempt("invalid-min", pAny, { minVersion: "TLSv9" }); await attempt("invalid-max", pAny, { maxVersion: 3 }); await attempt("conflict", pAny, { minVersion: "TLSv1.2", secureProtocol: "TLSv1_2_method" }); upTo12.close(); from13.close(); any.close(); + nullBounds.close(); } void main(); From 23ec584deb26ef1d50834d548abd0c1a7b463604 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 13:24:10 -0500 Subject: [PATCH 45/51] Read nullable union chunks of push and write by their representation Readable.push and Writable.write with a union chunk (Buffer, string and null arms) read the chunk's tag and payload from a tagged union box. A union with one reference arm and null, such as `string | null` or `Buffer | null`, is the arm pointer itself, so `push(line)` with `line: string | null` and `push(done ? null : text)` read past the string and crashed. Both now read the tag and payload with unionTag and unionPeek, as the other union consumers do. unshift and end refuse union chunks and are unchanged. --- .../compiler/src/backend/llvm/lib-stream.ts | 12 +-- .../6820-stream-push-nullable-chunks.ts.json | 6 ++ ...6821-stream-write-nullable-chunks.cjs.json | 6 ++ .../6820-stream-push-nullable-chunks.ts | 77 +++++++++++++++++++ .../6821-stream-write-nullable-chunks.cjs | 23 ++++++ .../6820-stream-push-nullable-chunks.ts.json | 1 + ...6821-stream-write-nullable-chunks.cjs.json | 1 + 7 files changed, 118 insertions(+), 8 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6820-stream-push-nullable-chunks.ts.json create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6821-stream-write-nullable-chunks.cjs.json create mode 100644 tests/corpus/6820-stream-push-nullable-chunks.ts create mode 100644 tests/corpus/6821-stream-write-nullable-chunks.cjs create mode 100644 tests/harness/corpus-costs/6820-stream-push-nullable-chunks.ts.json create mode 100644 tests/harness/corpus-costs/6821-stream-write-nullable-chunks.cjs.json diff --git a/packages/compiler/src/backend/llvm/lib-stream.ts b/packages/compiler/src/backend/llvm/lib-stream.ts index a319d0257..d3b4b1f5b 100644 --- a/packages/compiler/src/backend/llvm/lib-stream.ts +++ b/packages/compiler/src/backend/llvm/lib-stream.ts @@ -274,10 +274,9 @@ export function emitStreamLibCall(host: LlvmEmitterContext, e: LibCallExpr): LlV .filter((a) => a.tag >= 0); if (present.length === 0) throw new InternalCompilerError(`llvm emitter bug: ${e.fn} union lacks its arms`); - const tagP = B.tmp(); - const tag = B.tmp(); - B.line(`${tagP} = getelementptr inbounds %ScrUnion, ptr ${args[1]!.name}, i64 0, i32 1`); - B.line(`${tag} = load i32, ptr ${tagP}`); + // A tagged box, or the arm pointer itself for a nullable union + // (`string | null`, `Buffer | null`). + const tag = host.unionTag(args[1]!.name, t.unionId); const slot = B.slot(); B.entryAllocas.push(`${slot} = alloca i1`); const lj = B.newLabel("scu.j"); @@ -290,10 +289,7 @@ export function emitStreamLibCall(host: LlvmEmitterContext, e: LibCallExpr): LlV B.line(`store i1 ${r}, ptr ${slot}`); return; } - const pk = B.tmp(); - const pv = B.tmp(); - B.line(`${pk} = getelementptr inbounds %ScrUnion, ptr ${args[1]!.name}, i64 0, i32 5`); - B.line(`${pv} = load ptr, ptr ${pk} ; borrowed payload`); + const pv = host.unionPeek(args[1]!.name, t.unionId); // borrowed payload const r = B.tmp(); if (pushing) { host.declare(`declare zeroext i1 @${entry}(ptr, ptr)`); diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6820-stream-push-nullable-chunks.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6820-stream-push-nullable-chunks.ts.json new file mode 100644 index 000000000..26bb4ae9e --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6820-stream-push-nullable-chunks.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6820-stream-push-nullable-chunks.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6821-stream-write-nullable-chunks.cjs.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6821-stream-write-nullable-chunks.cjs.json new file mode 100644 index 000000000..e296beb66 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6821-stream-write-nullable-chunks.cjs.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6821-stream-write-nullable-chunks.cjs" + ], + "diags": [] +} diff --git a/tests/corpus/6820-stream-push-nullable-chunks.ts b/tests/corpus/6820-stream-push-nullable-chunks.ts new file mode 100644 index 000000000..5b79525b1 --- /dev/null +++ b/tests/corpus/6820-stream-push-nullable-chunks.ts @@ -0,0 +1,77 @@ +// Readable subclasses pushing `string | null` and `Buffer | null` chunks, a +// conditional `push(done ? null : chunk)`, and writes of `string | Buffer` +// chunks. +import { Readable, Writable } from "node:stream"; + +class Lines extends Readable { + private index = 0; + private readonly lines: string[]; + constructor(lines: string[]) { + super(); + this.lines = lines; + } + next(): string | null { + return this.index < this.lines.length ? this.lines[this.index++]! : null; + } + _read(): void { + const line: string | null = this.next(); + this.push(line); + } +} + +class Blocks extends Readable { + private sent = 0; + _read(): void { + const block: Buffer | null = this.sent < 3 ? Buffer.from([65 + this.sent, 10]) : null; + this.sent++; + this.push(block); + } +} + +class Countdown extends Readable { + private left: number; + constructor(left: number) { + super({ encoding: "utf8" }); + this.left = left; + } + _read(): void { + this.left--; + this.push(this.left < 0 ? null : `t-${this.left} `); + } +} + +function collect(label: string, stream: Readable): Promise { + return new Promise((resolve) => { + const parts: string[] = []; + stream.on("data", (chunk: Buffer | string) => + parts.push(typeof chunk === "string" ? chunk : chunk.toString()), + ); + stream.on("end", () => { + console.log(label, JSON.stringify(parts.join("")), parts.length); + resolve(); + }); + }); +} + +async function main(): Promise { + await collect("lines", new Lines(["a\n", "bb\n", "ccc\n"])); + await collect("empty lines", new Lines([])); + await collect("blocks", new Blocks()); + await collect("countdown", new Countdown(3)); + + const written: string[] = []; + const sink = new Writable({ + write(chunk: Buffer, _encoding, callback) { + written.push(chunk.toString()); + callback(); + }, + }); + const pieces: (string | Buffer)[] = ["x", Buffer.from("y"), "z"]; + for (const piece of pieces) { + const chunk: string | Buffer = piece; + console.log("write", Buffer.isBuffer(chunk) ? "buffer" : "string", sink.write(chunk)); + } + sink.end(() => console.log("written", JSON.stringify(written.join("")))); +} + +void main(); diff --git a/tests/corpus/6821-stream-write-nullable-chunks.cjs b/tests/corpus/6821-stream-write-nullable-chunks.cjs new file mode 100644 index 000000000..c7bf32bc0 --- /dev/null +++ b/tests/corpus/6821-stream-write-nullable-chunks.cjs @@ -0,0 +1,23 @@ +// Writes and pushes of a chunk that is a string or null: a null write throws +// ERR_STREAM_NULL_VALUES, and a null push ends the readable. +const { Writable, Readable } = require("node:stream"); +const written = []; +const sink = new Writable({ + write(chunk, _encoding, callback) { + written.push(chunk.toString()); + callback(); + }, +}); +for (let i = 0; i < 3; i++) { + const chunk = i === 1 ? null : `c${i}`; + try { + console.log("write", JSON.stringify(chunk), sink.write(chunk)); + } catch (error) { + console.log("write threw", error.name, error.code, error.message); + } +} +const r = new Readable({ read() {} }); +for (let i = 0; i < 3; i++) r.push(i === 2 ? null : `r${i}`); +r.on("data", (d) => console.log("data", String(d))); +r.on("end", () => console.log("end")); +sink.end(() => console.log("written", JSON.stringify(written.join("")))); diff --git a/tests/harness/corpus-costs/6820-stream-push-nullable-chunks.ts.json b/tests/harness/corpus-costs/6820-stream-push-nullable-chunks.ts.json new file mode 100644 index 000000000..dd2eae649 --- /dev/null +++ b/tests/harness/corpus-costs/6820-stream-push-nullable-chunks.ts.json @@ -0,0 +1 @@ +{ "plain": 3100, "san": 41100 } diff --git a/tests/harness/corpus-costs/6821-stream-write-nullable-chunks.cjs.json b/tests/harness/corpus-costs/6821-stream-write-nullable-chunks.cjs.json new file mode 100644 index 000000000..24b9052ff --- /dev/null +++ b/tests/harness/corpus-costs/6821-stream-write-nullable-chunks.cjs.json @@ -0,0 +1 @@ +{ "plain": 2400, "san": 40400 } From 1d6f26a071f35f64b9a5addf20428fa37dfa4c89 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 13:26:50 -0500 Subject: [PATCH 46/51] Give arrays of a known final length exact dense capacity --- .../src/backend/llvm/expr-containers.ts | 6 +- .../src/backend/llvm/expr-primitives.ts | 16 ++++- .../array-exact-capacity-builders.ts.json | 6 ++ packages/runtime/src/scr_array.c | 37 +++++++++++ packages/runtime/src/scr_array.h | 9 +++ tests/corpus/array-exact-capacity-builders.ts | 65 +++++++++++++++++++ .../array-exact-capacity-builders.ts.json | 1 + 7 files changed, 136 insertions(+), 4 deletions(-) create mode 100644 packages/compiler/test/ts7/baselines/order-parity/tests/corpus/array-exact-capacity-builders.ts.json create mode 100644 tests/corpus/array-exact-capacity-builders.ts create mode 100644 tests/harness/corpus-costs/array-exact-capacity-builders.ts.json diff --git a/packages/compiler/src/backend/llvm/expr-containers.ts b/packages/compiler/src/backend/llvm/expr-containers.ts index 8d1d1d28f..8970aa3be 100644 --- a/packages/compiler/src/backend/llvm/expr-containers.ts +++ b/packages/compiler/src/backend/llvm/expr-containers.ts @@ -647,10 +647,12 @@ export function emitArrIntrinsic( return { name: t, type: e.type }; } case "concatSpread": { + // Appends to concat's fresh result: its storage grows to exactly + // the combined length (scr_arr_concat_copy_exact). const src = emitBorrowedInput(host, e.args[0]!); - host.declare(`declare double @scr_arr_concat_copy(ptr, ptr)`); + host.declare(`declare double @scr_arr_concat_copy_exact(ptr, ptr)`); const t = B.tmp(); - B.line(`${t} = call double @scr_arr_concat_copy(ptr ${r.name}, ptr ${src.name})`); + B.line(`${t} = call double @scr_arr_concat_copy_exact(ptr ${r.name}, ptr ${src.name})`); return { name: t, type: e.type }; } case "unshift": { diff --git a/packages/compiler/src/backend/llvm/expr-primitives.ts b/packages/compiler/src/backend/llvm/expr-primitives.ts index 69f4def2e..cc6053f4b 100644 --- a/packages/compiler/src/backend/llvm/expr-primitives.ts +++ b/packages/compiler/src/backend/llvm/expr-primitives.ts @@ -855,11 +855,15 @@ export function emitContainerExpr( const elem = e.type.elem; const arr = B.tmp(); const small = host.appendedArrayLiterals.has(e); - const capacity = small ? 0 : (e.capacity ?? e.elems.length); + const spreadSet = new Set(e.spreads ?? []); + // With spreads, storage starts at the plain elements before the first + // spread, and each spread reserves exactly its source's length plus + // the plain elements up to the next one (scr_arr_reserve_spread). + const firstSpread = e.elems.findIndex((_, i) => spreadSet.has(i)); + const capacity = small ? 0 : (e.capacity ?? (firstSpread < 0 ? e.elems.length : firstSpread)); B.line(`${arr} = ${arrNewCall(host.shapeHost, elem, String(capacity), small)}`); const out = host.own({ name: arr, type: e.type }); const acc = elemAccess(elem); - const spreadSet = new Set(e.spreads ?? []); let pending: LlValue[] = []; let buffer: string | undefined; const flush = (): void => { @@ -874,6 +878,12 @@ export function emitContainerExpr( if (spreadSet.has(i)) { flush(); const v = host.emitExpr(el); + let plain = 0; + for (let j = i + 1; j < e.elems.length && !spreadSet.has(j); j++) plain++; + host.declare(`declare void @scr_arr_reserve_spread(ptr, ptr, ${host.sizeType})`); + B.line( + `call void @scr_arr_reserve_spread(ptr ${arr}, ptr ${v.name}, ${host.sizeType} ${plain})`, + ); host.emitArrayCopyLoop(arr, v.name, acc); return; } @@ -895,6 +905,8 @@ export function emitContainerExpr( const n = host.emitExpr(e.length); const arr = B.tmp(); B.line(`${arr} = ${arrNewCall(host.shapeHost, elem, "0")}`); + host.declare(`declare void @scr_arr_reserve_len(ptr, double)`); + B.line(`call void @scr_arr_reserve_len(ptr ${arr}, double ${n.name})`); const out = host.own({ name: arr, type: e.type }); const acc = elemAccess(elem); let fill = acc === "f64" ? f64Lit(0) : acc === "bool" ? "false" : "null"; diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/array-exact-capacity-builders.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/array-exact-capacity-builders.ts.json new file mode 100644 index 000000000..20dc485de --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/array-exact-capacity-builders.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/array-exact-capacity-builders.ts" + ], + "diags": [] +} diff --git a/packages/runtime/src/scr_array.c b/packages/runtime/src/scr_array.c index 51853afce..f8b5813a2 100644 --- a/packages/runtime/src/scr_array.c +++ b/packages/runtime/src/scr_array.c @@ -1470,6 +1470,14 @@ static void scr_arr_invalid_length(double length) { scr_throw_error_msg(SCR_ERR_RANGE, msg, sizeof msg - 1); } +static void scr_arr_reserve_exact(ScrArr *a, size_t need); + +/* A length set on an empty array without storage of its own (`new + * Array(n)`, Array.prototype.map's result) reserves exactly that many + * slots, as holes, up to this many; the writes that fill the array then + * need no geometric growth. */ +#define SCR_ARR_PRESIZE_LIMIT ((size_t)1 << 16) + void scr_arr_set_len(ScrArr *a, double length) { if (!(length >= 0) || !isfinite(length) || length != trunc(length) || length > (double)SCR_ARR_MAX_LENGTH) { @@ -1511,6 +1519,8 @@ void scr_arr_set_len(ScrArr *a, double length) { } if (scr_arr_ext_of(a)) scr_arr_ext_of(a)->sparse_len = scr_arr_sparse_lower_bound(scr_arr_ext_of(a)->sparse, scr_arr_ext_of(a)->sparse_len, next); } + } else if (a->len == 0 && next <= SCR_ARR_PRESIZE_LIMIT && (a->cap == 0 || scr_arr_inline(a))) { + scr_arr_reserve_exact(a, next); } a->len = next; } @@ -2037,6 +2047,33 @@ double scr_arr_concat_copy(ScrArr *a, const ScrArr *src) { return SCR_ARR_LEN_F64(a->len); } +/* Dense capacity of exactly `need` (rounded to the storage block) for an + * array whose final length is known: a fresh concat result, a spread + * literal, or a length set on an empty array. Arrays with sparse entries, + * and lengths past the dense cutoff, keep their ordinary growth. */ +static void scr_arr_reserve_exact(ScrArr *a, size_t need) { + if (need <= a->cap || need > SCR_ARR_DENSE_LIMIT || SCR_AX(a)->sparse_len != 0) return; + scr_arr_grow_dense_to(a, need, SCR_ARR_GROW_EXACT); +} + +double scr_arr_concat_copy_exact(ScrArr *a, const ScrArr *src) { + if (src != a && scr_arr_is_dense(src) && src->len <= SCR_ARR_MAX_LENGTH - a->len) + scr_arr_reserve_exact(a, (size_t)a->len + src->len); + return scr_arr_concat_copy(a, src); +} + +void scr_arr_reserve_spread(ScrArr *a, const ScrArr *src, size_t plain) { + if (src == a || !scr_arr_is_dense(src)) return; + size_t need = (size_t)a->len + src->len; + if (plain > SCR_ARR_DENSE_LIMIT || need > SCR_ARR_DENSE_LIMIT) return; + scr_arr_reserve_exact(a, need + plain); +} + +void scr_arr_reserve_len(ScrArr *a, double n) { + if (!(n >= 1) || n > (double)SCR_ARR_DENSE_LIMIT) return; + scr_arr_reserve_exact(a, (size_t)n); +} + /* ── unshift / reverse ────────────────────────────────────────────────── * Single-element unshift takes ownership, matching push. The emitter calls * it from right to left after evaluating every variadic argument, preserving diff --git a/packages/runtime/src/scr_array.h b/packages/runtime/src/scr_array.h index 95ee7ce00..2068a324d 100644 --- a/packages/runtime/src/scr_array.h +++ b/packages/runtime/src/scr_array.h @@ -442,6 +442,15 @@ double scr_arr_unshift_many(ScrArr *a, size_t count, const uint64_t *slots); double scr_arr_push_spread(ScrArr *a, const ScrArr *src); /* Append a concat argument by indexed property copy, preserving holes. */ double scr_arr_concat_copy(ScrArr *a, const ScrArr *src); +/* The same append for Array.prototype.concat's fresh result: dense storage + * grows to exactly the combined length instead of geometrically. */ +double scr_arr_concat_copy_exact(ScrArr *a, const ScrArr *src); +/* Exact dense capacity for an array literal about to copy `src` and then + * `plain` more elements (a spread position). */ +void scr_arr_reserve_spread(ScrArr *a, const ScrArr *src, size_t plain); +/* Exact dense capacity for `n` elements of an array that is filled to that + * length next (Array.from({ length: n })); no effect for other lengths. */ +void scr_arr_reserve_len(ScrArr *a, double n); /* unshift returns the new length and _ref takes ownership. The spread form * borrows a same-element-kind source, retains copied refs, and snapshots diff --git a/tests/corpus/array-exact-capacity-builders.ts b/tests/corpus/array-exact-capacity-builders.ts new file mode 100644 index 000000000..da3a587ed --- /dev/null +++ b/tests/corpus/array-exact-capacity-builders.ts @@ -0,0 +1,65 @@ +// Arrays whose final length is known while they are built: map results, +// new Array(n) filled by index, concat results, spread literals and +// Array.from({ length: n }). Holes, lengths and later appends behave as +// in JavaScript. +class Item { + id: number; + constructor(id: number) { + this.id = id; + } +} +const items: Item[] = []; +for (let i = 0; i < 9; i++) items.push(new Item(i)); +const ids = (xs: Item[]): string => xs.map((x) => x.id).join(","); + +const doubled = items.map((x) => new Item(x.id * 2)); +console.log(doubled.length, ids(doubled)); +doubled.push(new Item(99)); +console.log(doubled.length, ids(doubled)); + +const filled: Item[] = new Array(5); +for (let i = 0; i < filled.length; i++) filled[i] = items[i]!; +console.log(filled.length, ids(filled)); +const partial: number[] = new Array(6); +partial[1] = 10; +partial[4] = 40; +console.log(partial.length, 0 in partial, 1 in partial, JSON.stringify(partial)); +partial.push(7); +console.log(partial.length, JSON.stringify(partial)); + +const wide: number[] = []; +wide.length = 200000; +wide[150000] = 1; +console.log(wide.length, wide[150000], 5 in wide); +const empty: number[] = []; +empty.length = 0; +empty.push(3); +console.log(empty.length, empty[0]); + +const left = items.slice(0, 3); +const right = items.slice(3, 7); +const joined = left.concat(right, items.slice(8)); +console.log(joined.length, ids(joined)); +const holey: number[] = [1, 2, 3]; +holey.length = 5; +const withHoles = holey.concat([6, 7]); +console.log(withHoles.length, 3 in withHoles, 4 in withHoles, JSON.stringify(withHoles)); + +const spread = [new Item(-1), ...left, new Item(-2), ...right, new Item(-3)]; +console.log(spread.length, ids(spread)); +const spreadHoles = [...holey, 9]; +console.log(spreadHoles.length, 3 in spreadHoles, JSON.stringify(spreadHoles)); +const self = [...left, ...left]; +console.log(self.length, ids(self)); + +const zeros = Array.from({ length: 7 }, () => 0); +console.log(zeros.length, zeros.join("")); +const counted = Array.from({ length: 4 }, (_, i) => i * i); +console.log(counted.join(",")); +const none = Array.from({ length: 0 }, () => 1); +console.log(none.length); + +const sparseSource: number[] = []; +sparseSource[3] = 30; +const mapped = sparseSource.map((x) => x + 1); +console.log(mapped.length, 0 in mapped, 3 in mapped, mapped[3]); diff --git a/tests/harness/corpus-costs/array-exact-capacity-builders.ts.json b/tests/harness/corpus-costs/array-exact-capacity-builders.ts.json new file mode 100644 index 000000000..5a4cc1dfd --- /dev/null +++ b/tests/harness/corpus-costs/array-exact-capacity-builders.ts.json @@ -0,0 +1 @@ +{ "plain": 4230, "san": 6130 } From fc38e627915826a2e56e9f1843f3c5bc56df9752 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 14:10:23 -0500 Subject: [PATCH 47/51] Stop importing runtime bitcode into program builds --- docs/content/docs/cli.mdx | 2 +- native/llvm-codegen/README.md | 2 +- .../src/backend/native-codegen.test.ts | 24 ------ .../compiler/src/backend/native-codegen.ts | 8 -- packages/compiler/src/backend/optimization.ts | 4 +- .../compiler/src/backend/runtime-pack.test.ts | 79 +------------------ packages/compiler/src/backend/runtime-pack.ts | 46 ----------- packages/compiler/src/index.ts | 35 +------- 8 files changed, 11 insertions(+), 189 deletions(-) diff --git a/docs/content/docs/cli.mdx b/docs/content/docs/cli.mdx index daeb37ea0..efdb03d7d 100644 --- a/docs/content/docs/cli.mdx +++ b/docs/content/docs/cli.mdx @@ -127,7 +127,7 @@ With `--sparse`, the profile leaves out the functions the runs never reached. Th
Remove symbol and debug payload when linking an executable, including cross-compiled targets. This flag preserves code optimization.
--optimization <release|dev|speed>
-
Select native optimization for executable, assembly, or object output. The default is release (-O2), which emits string retains and the retains and releases of dynamic values inline and lets LLVM inline small runtime functions that keep no internal runtime state into executable code; programs large enough to be compiled in parallel partitions skip that inlining to keep build time and size in check. dev uses -O0 and emits source locations for TypeScript and JavaScript breakpoints and stack frames. LLVM builds include source variable names and native storage descriptions; see Native debugging for the supported representations. macOS executable builds also produce an adjacent .dSYM bundle. speed is -O2 plus optimizations that trade binary size and build time for run-time speed: it lets LLVM inline more small runtime functions into program code, emits the remaining reference counting inline, and on x86-64 starts every runtime function on a cache-line boundary. Expect larger executables and longer cold builds than release; how much faster a program runs depends on how much of its time is spent in the runtime. Runtime inlining needs a runtime pack built with the LLVM helper; where a runtime pack lacks it, programs link the same runtime without it and speed links the release runtime and keeps inline reference counting. Selecting speed never changes what release builds produce, and the two never share cached artifacts. Library profiles accept "optimization": "speed" for inline reference counting. In executables large enough to be divided into partitions, dev compiles each partition on its own, and release and speed add a step that lets partitions import small functions from each other; see Build parallelism.
+
Select native optimization for executable, assembly, or object output. The default is release (-O2), which emits string retains and the retains and releases of dynamic values inline. dev uses -O0 and emits source locations for TypeScript and JavaScript breakpoints and stack frames. LLVM builds include source variable names and native storage descriptions; see Native debugging for the supported representations. macOS executable builds also produce an adjacent .dSYM bundle. speed is -O2 plus optimizations that trade binary size and build time for run-time speed: it emits the remaining reference counting inline and, on x86-64, starts every runtime function on a cache-line boundary. Expect larger executables and longer cold builds than release; how much faster a program runs depends on how much of its time is spent in the runtime. Where a runtime pack has no speed runtime, speed links the release runtime and keeps inline reference counting. Selecting speed never changes what release builds produce, and the two never share cached artifacts. Library profiles accept "optimization": "speed" for inline reference counting. In executables large enough to be divided into partitions, dev compiles each partition on its own, and release and speed add a step that lets partitions import small functions from each other; see Build parallelism.
--windows-subsystem <console|gui>
Select the subsystem of a Windows executable. console is the default; gui prevents Windows from opening a console window for the app. Valid only for executable builds targeting Windows.
diff --git a/native/llvm-codegen/README.md b/native/llvm-codegen/README.md index 8debaeeaf..0e8abbc47 100644 --- a/native/llvm-codegen/README.md +++ b/native/llvm-codegen/README.md @@ -39,7 +39,7 @@ WebAssembly code generation differs from LLVM's default pipeline in two ways tha `--cache-dir ` names a private directory the caller owns for one helper build. Simplified partitions are stored under the hash of their exact source and objects under the hash of the exact module after importing, each with a trailing digest and published by rename, so a rebuild recompiles only the partitions an edit can affect. -Unsanitized `speed` program builds, and unsanitized `release` builds of programs small enough for one partition, pass `--import-bitcode ` once per runtime-pack unit the program links that ships bitcode. Small runtime functions reachable from the program's runtime calls are imported as `available_externally` so LLVM can inline them; the runtime objects remain the only definitions. The helper never runs a sanitizer pass, so with import it also drops the emitter's inert `sanitize_address` attributes, which would otherwise block that inlining. Without `--import-bitcode` the module is optimized exactly as emitted. +`--import-bitcode `, given once per runtime-pack unit that ships bitcode, imports runtime functions for inlining; scriptc's program builds do not pass it. Small runtime functions reachable from the program's runtime calls are imported as `available_externally` so LLVM can inline them; the runtime objects remain the only definitions. The helper never runs a sanitizer pass, so with import it also drops the emitter's inert `sanitize_address` attributes, which would otherwise block that inlining. Without `--import-bitcode` the module is optimized exactly as emitted. Runtime-pack builds emit each unit of the executable `speed` flavor through the helper: diff --git a/packages/compiler/src/backend/native-codegen.test.ts b/packages/compiler/src/backend/native-codegen.test.ts index 21c74e51c..545711a60 100644 --- a/packages/compiler/src/backend/native-codegen.test.ts +++ b/packages/compiler/src/backend/native-codegen.test.ts @@ -168,30 +168,6 @@ test("resolves a package helper, emits atomically, and caches by all native inpu expect(secondArtifact.dependencies).toEqual(firstArtifact.dependencies); }); -test("runtime bitcode imports reach the helper and key the native cache", async () => { - const pkg = await fakePackage(); - const base = request(pkg.root, pkg.packageJson, join(pkg.root, "plain.o")); - await emitNativeArtifact(base); - const imported = { - ...request(pkg.root, pkg.packageJson, join(pkg.root, "imported.o")), - importBitcode: { - paths: ["/pack/a.bc", "/pack/b.bc"], - digests: ["a".repeat(64), "b".repeat(64)], - }, - }; - await emitNativeArtifact(imported); - await emitNativeArtifact({ - ...imported, - outputPath: join(pkg.root, "changed.o"), - importBitcode: { ...imported.importBitcode, digests: ["a".repeat(64), "c".repeat(64)] }, - }); - await emitNativeArtifact({ ...imported, outputPath: join(pkg.root, "again.o") }); - const calls = (await readFile(pkg.log, "utf8")).trim().split("\n"); - expect(calls).toHaveLength(3); - expect(calls[0]).not.toContain("--import-bitcode"); - expect(calls[1]).toContain("--import-bitcode /pack/a.bc --import-bitcode /pack/b.bc"); -}); - test("profiles reach the helper and key the native cache by their bytes", async () => { const pkg = await fakePackage(); const base = request(pkg.root, pkg.packageJson, join(pkg.root, "plain.o")); diff --git a/packages/compiler/src/backend/native-codegen.ts b/packages/compiler/src/backend/native-codegen.ts index 39e570cd5..5b91c60e7 100644 --- a/packages/compiler/src/backend/native-codegen.ts +++ b/packages/compiler/src/backend/native-codegen.ts @@ -111,8 +111,6 @@ export interface NativeCodegenOptions { cacheRoot?: string | null; /** Program partitions, written to nativePartitionPaths(outputPath). */ partitions?: number; - /** Runtime bitcode whose small functions the helper imports for inlining. */ - importBitcode?: { paths: readonly string[]; digests: readonly string[] }; /** Profile-guided optimization: instrumentation or an indexed profile. */ profile?: NativeProfileOptions; } @@ -339,11 +337,6 @@ function cacheKey( .update("\0") .update(options.sourcePath) .update((options.partitions ?? 1) > 1 ? `\0partitions:${options.partitions}` : "") - .update( - options.importBitcode === undefined - ? "" - : `\0import:${options.importBitcode.digests.join(",")}`, - ) .update(options.profile === undefined ? "" : `\0${nativeProfileKey(options.profile)}`) .digest("hex"); } @@ -482,7 +475,6 @@ export async function emitNativeArtifact( "json", "--source-path", options.sourcePath, - ...(options.importBitcode?.paths ?? []).flatMap((path) => ["--import-bitcode", path]), ...(partitionCache === null ? [] : ["--cache-dir", partitionCache]), ...nativeProfileHelperArgs(options.profile), ]); diff --git a/packages/compiler/src/backend/optimization.ts b/packages/compiler/src/backend/optimization.ts index f26b99dcc..05e275afd 100644 --- a/packages/compiler/src/backend/optimization.ts +++ b/packages/compiler/src/backend/optimization.ts @@ -3,8 +3,8 @@ * - `release` (default): -O2 program code against the release runtime flavor. * - `dev`: -O0, source line tables, and cached object shards. * - `speed`: -O2 plus optimizations that trade binary size and build time for - * run-time speed (a size-for-speed runtime flavor, runtime bitcode import, - * inline reference counting). Release output never depends on them. + * run-time speed (a size-for-speed runtime flavor, inline reference + * counting). Release output never depends on them. * * Speed is an optimized posture everywhere release is: it shares -O2, release * linker flags, and release-only program partitioning. Cache keys spell it diff --git a/packages/compiler/src/backend/runtime-pack.test.ts b/packages/compiler/src/backend/runtime-pack.test.ts index 4fd53c01e..ee2301172 100644 --- a/packages/compiler/src/backend/runtime-pack.test.ts +++ b/packages/compiler/src/backend/runtime-pack.test.ts @@ -9,7 +9,6 @@ import type { NativeLinkFeatures } from "./native-link-info.js"; import { effectiveRuntimeFeatures, evaluateRuntimePredicate, - loadRuntimeBitcode, loadRuntimePack, parseRuntimePackManifest, RuntimePackError, @@ -262,80 +261,10 @@ describe("runtime pack manifests", () => { expect(fileHandle.archives.map((path) => path.split("/").at(-1))).toEqual(["zlib.a"]); }); - test("runtime bitcode follows the selected flavor variant and is verified", async () => { - const { root, packagePath, manifest } = await fixture(); - const bitcode = async (path: string, bytes: string) => { - await writeFile(join(root, path), bytes); - return { - path, - sha256: createHash("sha256").update(bytes).digest("hex"), - size: Buffer.byteLength(bytes), - }; - }; - const unit = manifest.flavors.release!.runtime_units[0]!; - const withBitcode: RuntimePackManifest = { - ...manifest, - flavors: { - ...manifest.flavors, - speed: { - optimization: "-O2", - runtime_units: [ - { - ...unit, - variants: [ - { ...unit.variants[0]!, bitcode: await bitcode("artifacts/base.bc", "base-bc") }, - { - ...unit.variants[1]!, - bitcode: await bitcode("artifacts/legacy.bc", "legacy-bc"), - }, - unit.variants[2]!, - ], - }, - ], - }, - }, - }; - const resolver = () => packagePath; - const speed = { target: MACOS_ARM64_TARGET, optimization: "speed" } as const; - // Release programs import their own units' bitcode. Speed programs - // never import release bitcode: without a speed flavor they import none. - await writeFile( - join(root, "runtime-pack.json"), - JSON.stringify({ - ...withBitcode, - flavors: { ...withBitcode.flavors, release: withBitcode.flavors.speed, speed: undefined }, - }), - ); - expect(await loadRuntimeBitcode({ ...speed, features: BASE, resolver })).toBeNull(); - const release = await loadRuntimeBitcode({ - target: MACOS_ARM64_TARGET, - optimization: "release", - features: BASE, - resolver, - }); - expect(release?.paths.map((path) => basename(path))).toEqual(["base.bc"]); - await writeFile(join(root, "runtime-pack.json"), JSON.stringify(withBitcode)); - const legacy = await loadRuntimeBitcode({ - ...speed, - features: { ...BASE, textDecoderLegacy: true }, - resolver, - }); - expect(legacy?.paths.map((path) => basename(path))).toEqual(["legacy.bc"]); - expect(legacy?.digests).toEqual([createHash("sha256").update("legacy-bc").digest("hex")]); - // A variant without bitcode imports nothing. - expect( - await loadRuntimeBitcode({ - ...speed, - features: { ...BASE, fileHandle: true }, - resolver, - }), - ).toBeNull(); - await writeFile(join(root, "artifacts/base.bc"), "damaged"); - await expect(loadRuntimeBitcode({ ...speed, features: BASE, resolver })).rejects.toThrow( - "hash mismatch", - ); - const malformed = structuredClone(withBitcode); - malformed.flavors.speed!.runtime_units[0]!.variants[0]!.bitcode = { + test("runtime unit bitcode must stay inside the pack", async () => { + const { manifest } = await fixture(); + const malformed = structuredClone(manifest); + malformed.flavors.release!.runtime_units[0]!.variants[0]!.bitcode = { path: "../escape.bc", sha256: "0".repeat(64), size: 1, diff --git a/packages/compiler/src/backend/runtime-pack.ts b/packages/compiler/src/backend/runtime-pack.ts index 5f813802c..2bfe71756 100644 --- a/packages/compiler/src/backend/runtime-pack.ts +++ b/packages/compiler/src/backend/runtime-pack.ts @@ -46,8 +46,6 @@ export interface RuntimePackSelection { sourceDependencies: NativeArtifactDependency[]; selectedRuntimeArtifacts: RuntimePackArtifact[]; selectedArchiveArtifacts: RuntimePackArtifact[]; - /** Bitcode of the selected runtime units that ship it, in unit order. */ - selectedRuntimeBitcode: RuntimePackArtifact[]; } async function verifyArtifact(root: string, artifact: RuntimePackArtifact): Promise { @@ -249,49 +247,5 @@ export async function loadRuntimePack(options: { sourceDependencies, selectedRuntimeArtifacts: selectedVariants, selectedArchiveArtifacts: selectedArchives, - selectedRuntimeBitcode: selectedVariants.flatMap((variant) => - variant.bitcode === undefined ? [] : [variant.bitcode], - ), - }; -} - -export interface RuntimeBitcodeImport { - paths: string[]; - /** Content digests, in path order, for cache keys. */ - digests: string[]; - dependencies: NativeArtifactDependency[]; -} - -/** Verified bitcode for the runtime units of the given flavor that a program - * links, for the helper's available_externally import. Release units carry - * their compiled bitcode reduced to the bodies that need no unit-local - * state; speed units carry their promoted bitcode. Null when the selected - * units ship no bitcode (a pack built without the helper, or no speed - * flavor). */ -export async function loadRuntimeBitcode(options: { - target: NativeTargetSpec; - features: NativeLinkFeatures; - optimization: "release" | "speed"; - env?: NodeJS.ProcessEnv; - resolver?: (specifier: string) => string; -}): Promise { - const pack = await loadRuntimePack(options); - // Without a speed flavor, speed programs link release units; only the - // speed flavor's promoted objects match speed bitcode. - if (options.optimization === "speed" && pack.manifest.flavors.speed === undefined) return null; - if (pack.selectedRuntimeBitcode.length === 0) return null; - const paths = await Promise.all( - pack.selectedRuntimeBitcode.map((artifact) => verifyArtifact(pack.root, artifact)), - ); - let dependencies: NativeArtifactDependency[]; - try { - dependencies = await snapshotDependencies([pack.manifestPath, ...paths]); - } catch { - throw new RuntimePackError("runtime pack bitcode changed during selection", "invalid"); - } - return { - paths, - digests: pack.selectedRuntimeBitcode.map((artifact) => artifact.sha256), - dependencies, }; } diff --git a/packages/compiler/src/index.ts b/packages/compiler/src/index.ts index 08457f86b..8b0ea6825 100644 --- a/packages/compiler/src/index.ts +++ b/packages/compiler/src/index.ts @@ -82,7 +82,7 @@ import { privateSiblingPath } from "./backend/build-cache.js"; import { nativeCodegenTarget, nativeCodegenTargetRefusal } from "./backend/targets.js"; import { windowsSubsystemLinkerArgs, type WindowsSubsystem } from "./backend/targets.js"; import { createNativeLinkInfo } from "./backend/native-link-info.js"; -import { RuntimePackError, loadRuntimeBitcode } from "./backend/runtime-pack.js"; +import { RuntimePackError } from "./backend/runtime-pack.js"; import { createNativeLinkPlan } from "./backend/link-plan.js"; import { nativeProfileKey, @@ -634,7 +634,6 @@ async function emitNativeProgramObject( entryPath: string, opts: CompileRequestOptions, llvm: LlvmInput, - features: EarlyExecutableNativeFeatures, profile: NativeProfileOptions | undefined, ): Promise { const stem = basename(entryPath).replace(/\.(ts|mts|cts|js|mjs|cjs)$/, ""); @@ -648,24 +647,6 @@ async function emitNativeProgramObject( const llvmBytes = llvmInputBytes(llvm); const partitions = opts.nativeProgramObject === true || target === null ? 1 : nativeProgramPartitions(llvmBytes); - // Optimized programs import small runtime functions from the bitcode of - // the exact runtime units they link, so LLVM can inline across the - // boundary. Release units ship only the bodies that need none of their - // unit-local state, which keeps the import cheap; speed units promote - // that state and offer more. Every partition reads the bitcode again, so - // release imports only into programs small enough for one partition: in - // larger ones the build-time and size cost outgrows the gain. - const runtimeBitcode = - opts.sanitize !== true && - target !== null && - (opts.optimization === "speed" || - (opts.optimization !== "dev" && nativeProgramPartitions(llvmBytes) === 1)) - ? await loadRuntimeBitcode({ - target, - features, - optimization: opts.optimization === "speed" ? "speed" : "release", - }) - : null; try { const artifact = await emitNativeArtifact({ outputPath: linkPath, @@ -675,16 +656,13 @@ async function emitNativeProgramObject( optimization: opts.optimization === "dev" ? "0" : "2", ...(opts.sanitize === undefined ? {} : { sanitize: opts.sanitize }), partitions, - ...(runtimeBitcode === null - ? {} - : { importBitcode: { paths: runtimeBitcode.paths, digests: runtimeBitcode.digests } }), ...(profile === undefined ? {} : { profile }), }); return { linkPath, partitionPaths: artifact.outputPaths.slice(1), artifactPath, - dependencies: [...artifact.dependencies, ...(runtimeBitcode?.dependencies ?? [])], + dependencies: artifact.dependencies, }; } catch (error) { await removeNativeProgramObject(linkPath, nativePartitionPaths(linkPath, partitions)); @@ -1467,7 +1445,6 @@ async function compileTracked( // A large program's LLVM exceeds the engine's string limit as one // string; the emitter takes it in parts, as on a cold build. await readTextParts(earlyHit.llvmPath), - earlyHit.native, profile, ); nativeInputPath = nativeProgramObject.linkPath; @@ -1583,13 +1560,7 @@ async function compileTracked( ); } try { - nativeProgramObject = await emitNativeProgramObject( - entryPath, - opts, - llvmSource, - nativeFeatures, - profile, - ); + nativeProgramObject = await emitNativeProgramObject(entryPath, opts, llvmSource, profile); timing("native-object"); await recordEarlyExecutableNativeBuild(cacheRoot, executableCacheOptions).catch( () => undefined, From 453bcf8e3887f33f9b47b6f07cdfc4266d38f6f1 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 14:19:48 -0500 Subject: [PATCH 48/51] Revisit only the changed sites of the optional-read fixed point --- .../compiler/src/frontend/lowering/lowerer.ts | 87 ++++++++++++++++--- .../lowering/runtime-optional-elements.ts | 22 ++++- .../lowering/site-dependencies.test.ts | 56 ++++++++++++ .../frontend/lowering/site-dependencies.ts | 79 +++++++++++++++++ 4 files changed, 232 insertions(+), 12 deletions(-) create mode 100644 packages/compiler/src/frontend/lowering/site-dependencies.test.ts create mode 100644 packages/compiler/src/frontend/lowering/site-dependencies.ts diff --git a/packages/compiler/src/frontend/lowering/lowerer.ts b/packages/compiler/src/frontend/lowering/lowerer.ts index cbd70a913..1b5e30545 100644 --- a/packages/compiler/src/frontend/lowering/lowerer.ts +++ b/packages/compiler/src/frontend/lowering/lowerer.ts @@ -121,6 +121,7 @@ import { ArrayElementStates } from "./runtime-optional-elements.js"; import { indexReadInBounds } from "./runtime-optional-bounds.js"; import { ArrayOwnership } from "./runtime-optional-ownership.js"; import { RecordFlowClasses } from "./runtime-optional-records.js"; +import { SiteDependencies } from "./site-dependencies.js"; import { StringIndexBounds } from "./string-index-bounds.js"; import { NanCodedReads } from "./nan-coded-reads.js"; import { RuntimeOptionalLocals } from "./runtime-optional-locals.js"; @@ -4688,10 +4689,15 @@ export class Lowerer { // resolveValueSymbol also flushes deferred diagnostics and applies // merged-namespace fences, whose authority remains ordinary lowering. this.checker.prefetchSymbolRoots(sourceFiles, true); + // The facts each fixed-point site reads, by the symbols, nodes and + // element keys they belong to (created with the site list below). + let siteDependencies: SiteDependencies | null = null; const symbolOf = (node: ts.Node): ts.Symbol | null => { const s = this.checker.getSymbolAtLocation(node); if (!s) return null; - return s.flags & ts.SymbolFlags.Alias ? this.checker.getAliasedSymbol(s) : s; + const symbol = s.flags & ts.SymbolFlags.Alias ? this.checker.getAliasedSymbol(s) : s; + siteDependencies?.read(symbol); + return symbol; }; const signatureOf = (node: ts.Identifier): FnSig | null => { const symbol = symbolOf(node); @@ -4876,7 +4882,10 @@ export class Lowerer { const callableSymbols = new Map(); const callableSymbolOf = (node: ts.Expression): ts.Symbol | null => { const cached = callableSymbols.get(node); - if (cached !== undefined) return cached; + if (cached !== undefined) { + if (cached !== null) siteDependencies?.read(cached); + return cached; + } let symbol = symbolOf(ts.isPropertyAccessExpression(node) ? node.name : node); if (symbol === null || signatureBySymbol.has(symbol)) { callableSymbols.set(node, symbol); @@ -5165,7 +5174,10 @@ export class Lowerer { if (ts.isPropertyAccessExpression(e) && isSettledIteratorValueRead(this, e)) return true; if (ts.isPropertyAccessExpression(e) && isOsConstantsSymbol(symbolOf(e.expression), this)) return true; - if (ts.isCallExpression(e) && this.runtimeOptionalReduceTypes.has(e)) return true; + if (ts.isCallExpression(e)) { + siteDependencies?.read(e); + if (this.runtimeOptionalReduceTypes.has(e)) return true; + } if ( ts.isCallExpression(e) && e.arguments.nodes.length === 0 && @@ -5663,6 +5675,7 @@ export class Lowerer { if (fnType?.kind === "func" && bodyReturnsOptional(fn)) { const widenedReturn = this.runtimeOptionalType(fnType.ret); provenance?.note("callback-return", fn, provenanceSite); + siteDependencies?.read(fn); const previousReturn = this.runtimeOptionalFunctionReturnType(fn, fnType.ret); this.runtimeOptionalFunctionReturns.set(fn, widenedReturn); if (!typeEquals(previousReturn, widenedReturn)) changed = true; @@ -5898,8 +5911,10 @@ export class Lowerer { arm.elem.kind === "union" ? this.armTag(arm.elem.unionId, UNDEFINED_T) >= 0 : isUnitType(arm.elem) || arm.elem.kind === "void" - ) + ) { + if (!optionalElementKeys.has(key)) siteDependencies?.changed(key); optionalElementKeys.add(key); + } keys.push(key); } return keys; @@ -6011,7 +6026,10 @@ export class Lowerer { return keys; }; const elementStates = new ArrayElementStates({ - elementAdmitsUndefined: (key) => optionalElementKeys.has(key), + elementAdmitsUndefined: (key) => { + siteDependencies?.read(key); + return optionalElementKeys.has(key); + }, ownerOf: (node) => ownership.ownerOf(node), allocationOwner: (node) => ownership.allocationOwner(node), arrayKeys: arrayKeysOf, @@ -6084,6 +6102,8 @@ export class Lowerer { note: provenance ? (fact, key, site) => provenance.note(`element-${fact}`, site, site, key) : null, + read: (fact) => siteDependencies?.read(fact), + changed: (fact) => siteDependencies?.changed(fact), }); // Each pass revisits every array callback site; the last visit holds // the decision at the fixed point. @@ -6385,6 +6405,7 @@ export class Lowerer { ) { if (promoteHofCallback(callback, [0, 1])) changed = true; const result = this.runtimeOptionalType(this.irTypeOf(node)); + siteDependencies?.read(node); const previous = this.runtimeOptionalReduceTypes.get(node); if (!previous || !typeEquals(previous, result)) { this.runtimeOptionalReduceTypes.set(node, result); @@ -6452,6 +6473,7 @@ export class Lowerer { ) continue; for (const target of familyBySymbol.get(symbol) ?? [symbol]) { + siteDependencies?.read(target); const shape = signatureBySymbol.get(target)?.params[i]; if (!shape || shape.mode !== "required") continue; const widened = addUndefined(shape.type); @@ -6496,6 +6518,7 @@ export class Lowerer { if (!ts.isSpreadElement(arg) && shape && i !== restAt && fieldsOfValue(arg)) { const fields = fieldsOfValue(arg)!; for (const target of familyBySymbol.get(symbol) ?? [symbol]) { + siteDependencies?.read(target); const parameterShape = signatureBySymbol.get(target)?.params[i]; const parameter = functionDeclBySymbol.get(target)?.parameters?.nodes[i]; if (!parameterShape || parameterShape.type.kind !== "record") continue; @@ -6541,6 +6564,7 @@ export class Lowerer { // argument can make its elements undefined, not the pack. // Keep every virtual implementation on the same array ABI. for (const target of familyBySymbol.get(symbol) ?? [symbol]) { + siteDependencies?.read(target); const rest = signatureBySymbol.get(target)?.params[i]; if (rest?.mode !== "rest" || rest.type.kind !== "array") continue; const widened = arrayOf(addUndefined(rest.type.elem)); @@ -6562,6 +6586,7 @@ export class Lowerer { // downstream calls and returns see the same optional value. const family = familyBySymbol.get(symbol); for (const target of family ?? [symbol]) { + siteDependencies?.read(target); const parameterShape = signatureBySymbol.get(target)?.params[i]; if (!parameterShape) continue; parameterShape.type = addUndefined(parameterShape.type); @@ -6592,8 +6617,21 @@ export class Lowerer { }); } }; - for (const node of sitesOf(sf)) scan(node); - return changed; + // A site whose facts are unchanged since its last visit would compute + // that visit's result again, except in a full pass. + const base = siteBase.get(sf)!; + const dependencies = siteDependencies!; + const sites = sitesOf(sf); + let fileChanged = false; + for (let i = 0; i < sites.length; i++) { + if (!fullPass && !dependencies.isStale(base + i)) continue; + changed = false; + dependencies.begin(base + i); + scan(sites[i]!); + dependencies.end(changed); + if (changed) fileChanged = true; + } + return fileChanged; }; const scanReturns = (symbol: ts.Symbol, decl: ts.FunctionLikeDeclaration): boolean => { let changed = false; @@ -6643,23 +6681,41 @@ export class Lowerer { optionalSymbols.add(symbol); this.earlyReadVars.add(symbol); } - let changed = true; - while (changed) { + // Number the sites of every file. The first and the last pass visit + // every site; the passes between visit only the sites whose facts + // changed (SiteDependencies). When such a pass changes nothing, a full + // pass confirms the fixed point, and the iteration resumes if it + // changes a fact after all. + const siteBase = new Map(); + let siteCount = 0; + for (const sf of sourceFiles) { + siteBase.set(sf, siteCount); + siteCount += sitesOf(sf).length; + } + siteDependencies = new SiteDependencies(siteCount); + let fullPass = true; + for (let changed = true, first = true; changed || !fullPass; first = false) { + fullPass = first || !changed; changed = false; for (const [symbol, field] of staticFieldsBySymbol) { if (!optionalSymbols.has(symbol) && mayBeOptional(field.initializer)) { optionalSymbols.add(symbol); + siteDependencies.changed(symbol); provenance?.note("static-field", field.initializer.parent, field.initializer); changed = true; } } for (const sf of sourceFiles) if (scanFile(sf)) changed = true; - for (const [symbol, decl] of functionDeclBySymbol) - if (scanReturns(symbol, decl)) changed = true; + for (const [symbol, decl] of functionDeclBySymbol) { + if (!scanReturns(symbol, decl)) continue; + siteDependencies.changed(symbol); + changed = true; + } const noteGenericReturn = (symbol: ts.Symbol, decl: ts.FunctionLikeDeclaration): void => { if (optionalReturns.has(symbol) || isJsSourceFile(ts.getSourceFileOfNode(decl))) return; if (!returnsOf(decl).some(genericMayBeOptional)) return; optionalReturns.add(symbol); + siteDependencies!.changed(symbol); this.runtimeOptionalGenericReturns.add(decl); provenance?.note("return", decl.name ?? decl, decl); changed = true; @@ -6696,6 +6752,7 @@ export class Lowerer { : previous; if (!typeEquals(previous, next)) { this.runtimeOptionalFunctionReturns.set(fn, next); + siteDependencies.changed(fn); provenance?.note("lambda-return", fn, fn.body); changed = true; } @@ -6713,10 +6770,12 @@ export class Lowerer { if (symbol && next.kind === "union" && this.armTag(next.unionId, UNDEFINED_T) >= 0) { if (!optionalReturns.has(symbol)) { optionalReturns.add(symbol); + siteDependencies.changed(symbol); changed = true; } } else if (symbol && !arithmeticReturns.has(symbol)) { arithmeticReturns.set(symbol, next); + siteDependencies.changed(symbol); changed = true; } } @@ -6733,6 +6792,7 @@ export class Lowerer { : this.runtimeOptionalType(previous); if (!typeEquals(previous, next)) { this.runtimeOptionalFunctionReturns.set(fn, next); + siteDependencies.changed(fn); changed = true; } } @@ -6746,6 +6806,7 @@ export class Lowerer { for (const symbol of optional ? family : []) { if (!optionalReturns.has(symbol)) { optionalReturns.add(symbol); + siteDependencies.changed(symbol); provenance?.note( "return", functionDeclBySymbol.get(symbol), @@ -6768,10 +6829,14 @@ export class Lowerer { const previous = arithmeticReturns.get(symbol); if (previous && typeEquals(previous, joined)) continue; arithmeticReturns.set(symbol, joined); + siteDependencies.changed(symbol); changed = true; } } } + // The site facts serve only the fixed point; release them for the rest + // of lowering. + siteDependencies = null; // NaN-coded slots convert optional values on entry instead of holding // the union, so promotion never applies to them. for (const symbol of [...optionalSymbols]) diff --git a/packages/compiler/src/frontend/lowering/runtime-optional-elements.ts b/packages/compiler/src/frontend/lowering/runtime-optional-elements.ts index 5ed011ce1..96d20d2c4 100644 --- a/packages/compiler/src/frontend/lowering/runtime-optional-elements.ts +++ b/packages/compiler/src/frontend/lowering/runtime-optional-elements.ts @@ -39,8 +39,16 @@ export interface ElementStateHost { allocationOwner(node: ts.Expression): ts.Symbol | null; /** Debug provenance for a new fact. */ note: ((fact: "undefined" | "holes", key: string, site: ts.Node | undefined) => void) | null; + /** The scan read, or added to, the facts of an element key, of an owning + * slot, or of every array (ANY_ARRAY), for dependency tracking + * (SiteDependencies). */ + read?(fact: string | ts.Symbol): void; + changed?(fact: string | ts.Symbol): void; } +/** The fact name of the facts shared by every array. */ +const ANY_ARRAY = "*"; + /** What an array value can contain besides present values. */ interface Contents { undefined: boolean; @@ -179,8 +187,10 @@ export class ArrayElementStates { if (built !== undefined) return built; } const owner = this.host.ownerOf(e); - if (owner !== null) + if (owner !== null) { + this.host.read?.(owner); return { undefined: this.undefinedOwners.has(owner), holes: this.holeyOwners.has(owner) }; + } return this.stored(this.host.arrayKeys(e)); } @@ -317,7 +327,9 @@ export class ArrayElementStates { /** The facts recorded for arrays with these element ABIs. */ private stored(keys: readonly string[] | null): Contents | null { if (this.ownFactsOnly) return NOTHING; + this.host.read?.(ANY_ARRAY); if (keys === null) return null; + for (const key of keys) this.host.read?.(key); return { undefined: this.anyUndefined || @@ -333,6 +345,7 @@ export class ArrayElementStates { const keys = this.host.arrayKeys(node); if (keys !== null && keys.length === 0) return false; const contents = this.contents(node); + this.host.read?.(ANY_ARRAY); return contents === null ? this.anyUndefined || this.anyHoley : contents.undefined || contents.holes; @@ -368,6 +381,7 @@ export class ArrayElementStates { this.holeyOwners.add(owner); changed = true; } + if (changed) this.host.changed?.(owner); return changed; } let changed = false; @@ -406,6 +420,7 @@ export class ArrayElementStates { if (owner === null) return this.addUndefined(this.host.arrayKeys(receiver)); if (this.undefinedOwners.has(owner)) return false; this.undefinedOwners.add(owner); + this.host.changed?.(owner); return true; } @@ -415,6 +430,7 @@ export class ArrayElementStates { if (owner === null) return this.addHoles(this.host.arrayKeys(receiver)); if (this.holeyOwners.has(owner)) return false; this.holeyOwners.add(owner); + this.host.changed?.(owner); return true; } @@ -422,6 +438,7 @@ export class ArrayElementStates { if (keys === null) { if (this.anyUndefined) return false; this.anyUndefined = true; + this.host.changed?.(ANY_ARRAY); this.host.note?.("undefined", "*", this.site); return true; } @@ -429,6 +446,7 @@ export class ArrayElementStates { for (const key of keys) { if (this.undefinedKeys.has(key)) continue; this.undefinedKeys.add(key); + this.host.changed?.(key); this.host.note?.("undefined", key, this.site); changed = true; } @@ -439,6 +457,7 @@ export class ArrayElementStates { if (keys === null) { if (this.anyHoley) return false; this.anyHoley = true; + this.host.changed?.(ANY_ARRAY); this.host.note?.("holes", "*", this.site); return true; } @@ -446,6 +465,7 @@ export class ArrayElementStates { for (const key of keys) { if (this.holeyKeys.has(key)) continue; this.holeyKeys.add(key); + this.host.changed?.(key); this.host.note?.("holes", key, this.site); changed = true; } diff --git a/packages/compiler/src/frontend/lowering/site-dependencies.test.ts b/packages/compiler/src/frontend/lowering/site-dependencies.test.ts new file mode 100644 index 000000000..ca5affcbe --- /dev/null +++ b/packages/compiler/src/frontend/lowering/site-dependencies.test.ts @@ -0,0 +1,56 @@ +import { expect, test } from "vitest"; +import { SiteDependencies } from "./site-dependencies.js"; + +test("a changed fact makes the sites that read it stale", () => { + const dependencies = new SiteDependencies(3); + const a = {}; + const b = "key"; + dependencies.begin(0); + dependencies.read(a); + dependencies.end(false); + dependencies.begin(1); + dependencies.read(b); + dependencies.end(false); + expect([0, 1, 2].map((site) => dependencies.isStale(site))).toEqual([false, false, false]); + dependencies.changed(b); + expect([0, 1, 2].map((site) => dependencies.isStale(site))).toEqual([false, true, false]); + dependencies.begin(1); + expect(dependencies.isStale(1)).toBe(false); +}); + +test("a visit that changes a fact makes every reader of its facts stale", () => { + const dependencies = new SiteDependencies(3); + const shared = {}; + const own = {}; + dependencies.begin(0); + dependencies.read(shared); + dependencies.end(false); + dependencies.begin(2); + dependencies.read(own); + dependencies.end(false); + dependencies.begin(1); + dependencies.read(shared); + dependencies.read(shared); + dependencies.end(true); + expect([0, 1, 2].map((site) => dependencies.isStale(site))).toEqual([true, true, false]); +}); + +test("reads outside a visit are not recorded", () => { + const dependencies = new SiteDependencies(1); + const fact = {}; + dependencies.read(fact); + dependencies.changed(fact); + expect(dependencies.isStale(0)).toBe(false); +}); + +test("a site visited again is listed once per fact", () => { + const dependencies = new SiteDependencies(3); + const fact = {}; + for (const site of [0, 2, 1, 2, 0]) { + dependencies.begin(site); + dependencies.read(fact); + dependencies.end(false); + } + dependencies.changed(fact); + expect([0, 1, 2].map((site) => dependencies.isStale(site))).toEqual([true, true, true]); +}); diff --git a/packages/compiler/src/frontend/lowering/site-dependencies.ts b/packages/compiler/src/frontend/lowering/site-dependencies.ts new file mode 100644 index 000000000..da8531e48 --- /dev/null +++ b/packages/compiler/src/frontend/lowering/site-dependencies.ts @@ -0,0 +1,79 @@ +/** The facts each site of a fixed-point scan read on its last visit, so that + * a pass visits again only the sites whose facts changed since. + * + * Sites are numbered from 0. A fact is any value usable as a Map key: a + * symbol, a syntax node, or a string that names a derived fact. A visit + * reports what it reads (`read`). When a visit changes a fact, every site + * that read one of the facts this visit read is visited again: the facts a + * visit changes belong to what it read. Facts changed outside a visit, or + * not read before they change, are reported with `changed`. + * + * A site that is not visited again would compute what its last visit + * computed, as long as every fact it reads is reported. The scan confirms + * that with a pass over every site once a pass changes nothing, and resumes + * when that pass changes a fact: a fact this tracking misses costs passes, + * not a different result. */ +export class SiteDependencies { + /** Each fact's entry: the visit that last read it, so that a visit lists a + * fact once, followed by the sites that read it, ascending. */ + private readonly readers = new Map(); + private readonly stale: Uint8Array; + private site = -1; + private visit = 0; + private facts: unknown[] = []; + + constructor(siteCount: number) { + this.stale = new Uint8Array(siteCount); + } + + /** A fact that site `index` read changed since its last visit. */ + isStale(index: number): boolean { + return this.stale[index] === 1; + } + + /** Starts a visit of site `index`. */ + begin(index: number): void { + this.site = index; + this.visit++; + this.stale[index] = 0; + this.facts = []; + } + + /** Ends the visit; `changed` when it changed a fact. */ + end(changed: boolean): void { + this.site = -1; + if (changed) for (const fact of this.facts) this.changed(fact); + } + + /** The site being visited read `fact`. */ + read(fact: unknown): void { + if (this.site < 0) return; + const entry = this.readers.get(fact); + if (entry === undefined) { + this.readers.set(fact, [this.visit, this.site]); + this.facts.push(fact); + return; + } + if (entry[0] === this.visit) return; + entry[0] = this.visit; + this.facts.push(fact); + // A pass visits sites in ascending order, so a site is new at the end; + // a site visited again is usually listed already. + const site = this.site; + let low = 1; + let high = entry.length; + if (entry[high - 1]! < site) low = high; + while (low < high) { + const middle = (low + high) >>> 1; + if (entry[middle]! < site) low = middle + 1; + else high = middle; + } + if (entry[low] !== site) entry.splice(low, 0, site); + } + + /** `fact` changed: every site that read it is visited again. */ + changed(fact: unknown): void { + const entry = this.readers.get(fact); + if (entry !== undefined) for (let i = 1; i < entry.length; i++) this.stale[entry[i]!] = 1; + } +} From 558fb7c8bab2c969592df5c0ff8224a8d7cc56b6 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 14:20:20 -0500 Subject: [PATCH 49/51] Bind only conditional-dependent class type arguments as checker types A generic class instance now binds a type parameter to its checker type only when a conditional type in the class depends on it, and that checker type keys the instance. Before, every type argument was bound while only literal arguments of those parameters were in the key, so two instances such as Holder and Holder with a `T[K]` field shared the first instance's layout. The other parameters get no checker binding, so member types that depend on their literal identity stay unresolved, as before. Detection now follows the type aliases and interfaces a member's type refers to, so a conditional wrapped in a union alias, such as `(K extends "required" ? T : T | undefined) | null`, also separates the instances. --- .../src/frontend/lowering/lower-classes.ts | 77 ++++++++++++++----- .../6908-generic-class-literal-conditional.ts | 15 ++++ 2 files changed, 71 insertions(+), 21 deletions(-) diff --git a/packages/compiler/src/frontend/lowering/lower-classes.ts b/packages/compiler/src/frontend/lowering/lower-classes.ts index ae03a748c..3db208f55 100644 --- a/packages/compiler/src/frontend/lowering/lower-classes.ts +++ b/packages/compiler/src/frontend/lowering/lower-classes.ts @@ -4616,22 +4616,28 @@ export function genericClassInstanceType( if (bound) tsArgs.push(bound); } } - // A literal argument for a type parameter that a conditional type - // depends on (`K extends "required" ? T : T | undefined`) selects the - // members' types, so it keys its own instance; mapType alone widens it. + // A type parameter that a conditional type in the class depends on + // (`K extends "required" ? T : T | undefined`) is bound to its checker + // type for the members, and that type keys the instance: mapType alone + // widens a literal, so Field and Field would otherwise share one layout. The other parameters get + // no checker binding, so nothing in the members depends on a literal + // the key does not carry. gci.literalSensitive ??= literalSensitiveTypeParams(lowerer, gci); - const literalKey = (i: number): string | null => { - if (!gci.literalSensitive!.has(gci.typeParams[i]!)) return null; - const arg = tsArgs[i]; + const sensitive = (i: number): boolean => + tsArgs.length === gci.typeParams.length && gci.literalSensitive!.has(gci.typeParams[i]!); + const checkerKey = gci.typeParams + .map((_, i) => (sensitive(i) ? `#${tsArgs[i]!.id}` : "*")) + .join(","); + const key = + mapped.map(typeKey).join(",") + (gci.literalSensitive.size > 0 ? `@${checkerKey}` : ""); + const literalText = (i: number): string | null => { + const arg = sensitive(i) ? tsArgs[i] : undefined; if (arg?.isStringLiteralType()) return JSON.stringify(arg.value); if (arg?.isNumberLiteralType()) return String(arg.value); if (arg && arg.flags & ts.TypeFlags.BooleanLiteral) return lowerer.checker.typeToString(arg); return null; }; - const literals = gci.typeParams.map((_, i) => literalKey(i)); - const key = - mapped.map(typeKey).join(",") + - (literals.some((l) => l !== null) ? `@${literals.map((l) => l ?? "*").join(",")}` : ""); const existing = gci.instances.get(key); if (existing) { if (existing.info) lowerer.noteGenericClassInstanceDemand(existing.info); @@ -4654,9 +4660,10 @@ export function genericClassInstanceType( const bindings = new Map(); gci.typeParams.forEach((tp, i) => bindings.set(tp, mapped[i]!)); const tsBindings = new Map(); - if (tsArgs.length === gci.typeParams.length) - gci.typeParams.forEach((tp, i) => tsBindings.set(tp, tsArgs[i]!)); - const rendered = mapped.map((m, i) => literals[i] ?? lowerer.fmt(m)).join(", "); + gci.typeParams.forEach((tp, i) => { + if (sensitive(i)) tsBindings.set(tp, tsArgs[i]!); + }); + const rendered = mapped.map((m, i) => literalText(i) ?? lowerer.fmt(m)).join(", "); const typeArgsText = `<${rendered.length > 80 ? rendered.slice(0, 77) + "..." : rendered}>`; const prevBindings = lowerer.typeParamBindings; const prevTsBindings = lowerer.typeParamTsBindings; @@ -4702,12 +4709,43 @@ export function genericClassInstanceType( /** The class's type parameters that a conditional type inside it depends * on: the parameters named within a type node that is a conditional type - * or refers to one (`get(): Value` with `type Value = K - * extends "required" ? T : T | undefined`). Their literal arguments pick - * the conditional's branch. */ + * or reaches one through the type aliases and interfaces it refers to + * (`get(): Value` with `type Value = K extends "required" ? T + * : T | undefined`, also when the alias wraps the conditional in a union + * or another alias). Their arguments pick the conditional's branch. */ function literalSensitiveTypeParams(lowerer: Lowerer, gci: GenericClassInfo): Set { const params = new Set(gci.typeParams); const found = new Set(); + const reaches = new Map(); + const declarationsReach = (symbol: ts.Symbol): boolean => { + const target = + symbol.flags & ts.SymbolFlags.Alias ? lowerer.checker.getAliasedSymbol(symbol) : symbol; + const known = reaches.get(target); + if (known !== undefined) return known; + reaches.set(target, false); // a recursive alias reaches through its other parts + const result = target.declarations.some((d) => + ts.isTypeAliasDeclaration(d) + ? reachesConditional(d.type) + : ts.isInterfaceDeclaration(d) + ? d.members.nodes.some(reachesConditional) + : false, + ); + reaches.set(target, result); + return result; + }; + const reachesConditional = (node: ts.Node): boolean => { + if (ts.isConditionalTypeNode(node)) return true; + if (ts.isTypeReferenceNode(node)) { + const name = ts.isIdentifier(node.typeName) ? node.typeName : node.typeName.right; + const symbol = lowerer.checker.getSymbolAtLocation(name); + if (symbol && !params.has(symbol) && declarationsReach(symbol)) return true; + } + let found = false; + ts.forEachChild(node, (child) => { + if (!found && reachesConditional(child)) found = true; + }); + return found; + }; const collect = (node: ts.Node): void => { if (ts.isTypeReferenceNode(node) && ts.isIdentifier(node.typeName)) { const symbol = lowerer.checker.getSymbolAtLocation(node.typeName); @@ -4716,12 +4754,9 @@ function literalSensitiveTypeParams(lowerer: Lowerer, gci: GenericClassInfo): Se ts.forEachChild(node, collect); }; const visit = (node: ts.Node): void => { - if ( - ts.isConditionalTypeNode(node) || - (ts.isTypeReferenceNode(node) && - lowerer.checker.getTypeFromTypeNode(node).isConditionalType()) - ) { + if (ts.isTypeNode(node) && reachesConditional(node)) { collect(node); + return; } ts.forEachChild(node, visit); }; diff --git a/tests/corpus/6908-generic-class-literal-conditional.ts b/tests/corpus/6908-generic-class-literal-conditional.ts index d06669be3..8bf7a0533 100644 --- a/tests/corpus/6908-generic-class-literal-conditional.ts +++ b/tests/corpus/6908-generic-class-literal-conditional.ts @@ -48,3 +48,18 @@ class Switch { } } console.log(new Switch(true).label(), new Switch(false).label()); + +// A conditional wrapped in a union alias selects its branch the same way. +type MaybeValue = (K extends "required" ? T : T | undefined) | null; +class Slot { + value: MaybeValue; + kind: K; + constructor(value: MaybeValue, kind: K) { + this.value = value; + this.kind = kind; + } +} +const filled = new Slot("a", "required"); +const empty = new Slot(undefined, "optional"); +const cleared = new Slot(null, "optional"); +console.log(filled.value, empty.value, cleared.value, filled.kind, empty.kind); From 3b35db6b6bee6c3a0785cea52b0952108d8f632a Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 14:24:57 -0500 Subject: [PATCH 50/51] Resolve the stream-construction fixture's file read on close readFileStream resolved on 'end' while its 'close' listener printed "released". 'close' follows 'end' once the descriptor is closed, so the line landed wherever that completion fell among the child processes the fixture starts next, and the comparison with Node depended on timing. The read now resolves from its 'close' listener, after the line is printed. --- tests/fixtures/node-types/stream-construction.ts | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/fixtures/node-types/stream-construction.ts b/tests/fixtures/node-types/stream-construction.ts index a28643f7c..dfe4a2320 100644 --- a/tests/fixtures/node-types/stream-construction.ts +++ b/tests/fixtures/node-types/stream-construction.ts @@ -42,17 +42,21 @@ function gunzipThrough(payload: Buffer): Promise { }); } +// Resolves after 'close', which follows 'end' once the descriptor is +// closed: the result never races the child processes started next. function readFileStream(file: string): Promise { return new Promise((resolve) => { const stream = fs.createReadStream(file); let text = ""; - const release = () => console.log("released"); + const release = () => { + console.log("released"); + resolve(text); + }; stream.on("close", release); stream.on("error", release); stream.on("data", (chunk) => { text += chunk.toString(); }); - stream.on("end", () => resolve(text)); }); } From c1b413e5ccfe14f3375dbfad85172cbcda0e0f08 Mon Sep 17 00:00:00 2001 From: Chris Tate Date: Wed, 21 Oct 2026 14:25:23 -0500 Subject: [PATCH 51/51] Shorten the cycle collector's per-object paths markGray's edge visitor kept a stack frame on every call, though most edges end at an object that is skipped or already gray. Those now return without one, and the first visit and skipped cross-generation edges tail-call out of line, as the scan visitors already do. The walks no longer call the dense-array walker for an empty array. The routing of nursery candidates past bypassed levels inlines its bypass test, and the end of each pass tests its work vectors' capacity inline. What is collected, and when, is unchanged. --- packages/runtime/src/scr_cycle.c | 77 ++++++++++++++++++++------------ 1 file changed, 49 insertions(+), 28 deletions(-) diff --git a/packages/runtime/src/scr_cycle.c b/packages/runtime/src/scr_cycle.c index 682f8bfcd..591f544b8 100644 --- a/packages/runtime/src/scr_cycle.c +++ b/packages/runtime/src/scr_cycle.c @@ -991,7 +991,8 @@ static inline ScrCycKind *scr_cyc_kind(uint32_t kind) { /* Whether a candidate of this type at level `gen` (nursery or mature) * skips that level's passes (see above). The last candidate a bypass sends * past the level opens a probe window, with fresh counts. */ -static inline bool scr_cyc_bypassed(const ScrCycType *type, unsigned gen) { +static inline __attribute__((always_inline)) bool scr_cyc_bypassed(const ScrCycType *type, + unsigned gen) { uint32_t kind = type->kind; if (kind >= scr_cyc_kinds_cap) return false; ScrCycLevelStat *k = &scr_cyc_kinds[kind].level[gen]; @@ -1006,8 +1007,10 @@ static inline bool scr_cyc_bypassed(const ScrCycType *type, unsigned gen) { } /* The generation a release buffers an object into: its own, or past the - * levels its type bypasses. */ -static inline unsigned scr_cyc_buffer_gen(const ScrCycType *type, unsigned gen) { + * levels its type bypasses. Both run once per candidate that the routing + * below moves, so they are inlined there too. */ +static inline __attribute__((always_inline)) unsigned scr_cyc_buffer_gen(const ScrCycType *type, + unsigned gen) { if (gen == SCR_CYC_NURSERY && scr_cyc_bypassed(type, SCR_CYC_NURSERY)) gen = SCR_CYC_MATURE; if (gen == SCR_CYC_MATURE && scr_cyc_bypassed(type, SCR_CYC_MATURE)) gen = SCR_CYC_OLD; return gen; @@ -1236,13 +1239,17 @@ void scr_cyc_on_release(void *obj) { } /* Visit every child of `obj` with VISIT, through ARRAY_WALK for a dense - * cycle-capable array and through the type's trace otherwise. */ + * cycle-capable array and through the type's trace otherwise. An empty + * dense array has no children to visit, and ARRAY_WALK is not called. */ #define SCR_CYC_CHILDREN(obj, type, ARRAY_WALK, VISIT, depth) \ - do { \ - if ((type)->kind == SCR_CYC_KIND_ARRAY && !(((const ScrArr *)(obj))->spare & SCR_ARR_HAS_EXT)) \ - ARRAY_WALK((const ScrArr *)(obj), (void *)(depth)); \ - else \ - (type)->trace((obj), (VISIT), (void *)(depth)); \ + do { \ + if ((type)->kind == SCR_CYC_KIND_ARRAY && \ + !(((const ScrArr *)(obj))->spare & SCR_ARR_HAS_EXT)) { \ + if (((const ScrArr *)(obj))->len != 0) \ + ARRAY_WALK((const ScrArr *)(obj), (void *)(depth)); \ + } else { \ + (type)->trace((obj), (VISIT), (void *)(depth)); \ + } \ } while (0) static void scr_mg_visit(void *child, void *ctx); @@ -1253,12 +1260,23 @@ SCR_CYC_ARRAY_WALK(scr_sb_array, scr_sb_visit) SCR_CYC_ARRAY_WALK(scr_scan_array, scr_scan_visit) static inline __attribute__((always_inline)) void scr_mark_gray(void *obj, uintptr_t depth); +/* An edge into a generation this pass does not walk, for settle. */ +static __attribute__((noinline)) void scr_mg_skip_edge(void *child) { + scr_cyc_push_work(&scr_xgen_edges, scr_gray_holder); + scr_cyc_push_work(&scr_xgen_edges, child); +} +/* The first visit of an object: gray it and walk its children. */ +static __attribute__((noinline)) void scr_mg_first(void *child, void *ctx) { + scr_mark_gray(child, (uintptr_t)ctx); +} +/* Most edges end at an object that is skipped or gray already. Those + * return without a stack frame; the other two cases tail-call out of + * line. */ static void scr_mg_visit(void *child, void *ctx) { if (child == NULL || SCR_RC(child) == SCR_RC_IMMORTAL) return; ScrCycHdr *h = scr_cyc_hdr(child); if (SCR_CYC_WALK_GEN(h) > scr_gen_limit) { - scr_cyc_push_work(&scr_xgen_edges, scr_gray_holder); - scr_cyc_push_work(&scr_xgen_edges, child); + scr_mg_skip_edge(child); return; } size_t rc = --SCR_RC(child); @@ -1269,7 +1287,7 @@ static void scr_mg_visit(void *child, void *ctx) { } return; } - scr_mark_gray(child, (uintptr_t)ctx); + scr_mg_first(child, ctx); } /* markGray reaches `obj` (its count already trial-decremented): gray it, * number it with the walk's component, record it, count it while it is @@ -1436,6 +1454,25 @@ static SCR_CYC_COMPACT void scr_xg_release(void *obj) { scr_rc_destroy(obj, scr_xg_destroy); } +/* A work vector that holds more than this many entries is freed at the end + * of a pass that used less than a quarter of it, so a spike in one pass + * does not keep its buffers for the rest of the program, while passes of a + * steady size keep reusing theirs. */ +#define SCR_CYC_KEEP_VECTOR 65536 + +static SCR_CYC_COMPACT void scr_cyc_trim_free(ScrVec *vec) { + free(vec->v); + vec->v = NULL; + vec->cap = 0; +} +/* Every pass ends by trimming its work vectors, and most of them stay + * small: only the capacity test runs inline. */ +static inline void scr_cyc_trim(ScrVec *vec) { + if (vec->cap > SCR_CYC_KEEP_VECTOR && vec->n == 0 && vec->used < vec->cap / 4) + scr_cyc_trim_free(vec); + vec->used = 0; +} + #ifdef SCR_YIN /* ── young-in nursery ─────────────────────────────────────────────────── */ @@ -1488,7 +1525,6 @@ static SCR_CYC_COMPACT void scr_xg_release(void *obj) { * deletion reaches the survivor from the root's candidate; garbage formed * after the promotion has a candidate from the release that formed it, * since survivors are no longer young. */ -static void scr_cyc_trim(ScrVec *vec); #define SCR_YIN_DEFAULT_SIZE 4096 #define SCR_CYC_YIN_PROMOTED SCR_CYC_NURSERY @@ -1928,21 +1964,6 @@ static SCR_CYC_COMPACT size_t scr_cyc_settle(unsigned gen_limit, bool doomed) { return freed; } -/* A work vector that holds more than this many entries is freed at the end - * of a pass that used less than a quarter of it, so a spike in one pass - * does not keep its buffers for the rest of the program, while passes of a - * steady size keep reusing theirs. */ -#define SCR_CYC_KEEP_VECTOR 65536 - -static SCR_CYC_COMPACT void scr_cyc_trim(ScrVec *vec) { - if (vec->n == 0 && vec->cap > SCR_CYC_KEEP_VECTOR && vec->used < vec->cap / 4) { - free(vec->v); - vec->v = NULL; - vec->cap = 0; - } - vec->used = 0; -} - /* The end of every pass. The candidate list keeps its count, which the * scheduler reads; its buffer goes when this pass's candidates filled * less than a quarter of it. */