diff --git a/CHANGELOG.md b/CHANGELOG.md index 612df47b5..e8ab725f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,42 @@ All notable changes to scriptc will be documented in this file. +## 0.4.5 + +### Highlights + +- **Fixes:** + - Release builds read a typed array's length without a header load that could read a freed view. + - `net.Socket` `write()` returns Node's backpressure boolean and fires `'drain'`. + - Super calls in JavaScript method values call the declared base method. + - Calling a non-function through an asserted callee throws Node's TypeError. +- **Faster programs:** + - Exceptions propagate by unwinding from directly called functions on Linux, instead of status checks after every call. + - Typed-array loops prove their bounds and keep their views disjoint in more cases, including views over shared memory in worker programs; integer loops stay in integers across nested counted loops, pure helper calls, powers of two and byte reads. + - Lone array callbacks run as loops with their bodies inlined, and row sums run two rows per pass. + - Number-key maps look keys up on their integer view and write updates through the entry the read found; small number-key maps use a direct index. + - Strings: base64 decodes eight characters at a time; Buffers fill, swap byte order and decode `utf16le` a word at a time; readline splits input in one scan; string scans over non-ASCII strings use their UTF-16 units. + - Untyped code: property caches also cache absent properties, object literal members go through the store caches, calls on plain arrays skip the method ladder, and dynamic values dispose without key-cache walks. + - JSON parses into discriminated unions, and dynamic values convert into nested unions, in one pass. + - BigInt division settles quotients from the top limbs, and limb shifts are vectorized. +- **Faster builds:** + - Function bodies are emitted on up to eight workers, including in dev builds, and repeated work in LLVM module emission is cut. + - Linux executables link with lld when the driver finds it, and unchanged executables are reused without selecting the runtime pack. + - Dev builds reuse their frontend output after edits that keep token positions, and unchanged frontend inputs are checked by file identity before hashing. + - Lowering fetches only the syntax trees it reads, and the large compiler tables are built on first use. + - Release builds of the native compiler use a profile collected on the release commit. + - Type checking can use a tsc-ts server shared by later builds (opt-in). +- **Node.js parity:** + - Crypto: RSA key generation lengths, RSA-PSS keys, encrypted private keys, `X509Certificate` fields and methods, and TLS certificates and `tls.Server` contexts. + - zlib: Brotli compression and decompression. + - Dates are mutable objects with Node's setters and conversions; URL hosts and paths parse as the URL Standard does, including non-ASCII hosts with UTS #46. + - `Math`, `JSON` and `Reflect` are real objects in JavaScript; objects can inherit from JavaScript class prototypes; typed arrays held as untyped values support named properties and `Symbol.species`. + - Error messages: `JSON.parse` syntax errors use V8's messages and positions; `Object.getPrototypeOf` of an Error answers `Error.prototype`. + - `util.inspect` of dynamic classes, functions and `%o` arguments, and `Symbol.toStringTag` fields and getters, match Node. +- **npm packages:** more npm and CommonJS dependencies compile statically, including namespace-style packages with guarded declarations and `var` exports, and untyped Maps, Sets and arrays are typed from their uses. + + + ## 0.4.4 ### Highlights @@ -29,8 +65,6 @@ All notable changes to scriptc will be documented in this file. - Split CommonJS `require` bindings, `__`-prefixed exports, string-key destructuring of `any` values, and record key order for values JavaScript built. - **npm packages:** zod 4 and `@hono/zod-validator` values are projected at the declaration boundary. - - ## 0.4.3 ### Highlights diff --git a/benchmarks/runtime/README.md b/benchmarks/runtime/README.md index af3d3d239..0157503fd 100644 --- a/benchmarks/runtime/README.md +++ b/benchmarks/runtime/README.md @@ -32,6 +32,7 @@ Application-shaped workloads for measuring the speed of compiled executables. Ea | `regex-logs` | cpu | regex `exec` with groups, `test`, global `replace` | | `regex-replacers` | cpu | regex `replace`/`replaceAll` with inline callbacks whose parameters have no annotations: dictionary placeholders, snake_case to camelCase, date reordering, optional captures, capitalization, and a string search that reads the offset | | `region-builds` | cpu | a watch-mode build loop over `LifetimeRegion`: each build binds a program in a new region and releases the previous one, which a cache keeps referenced for two builds | +| `string-iteration` | cpu | code point iteration over text that mixes ASCII, Latin-1 letters, CJK and emoji: a for-of word tokenizer, spread reversal, and `Array.from` over words | | `template-render` | cpu | template literals, escaping, `map`/`join` | | `throw-chains` | cpu | exceptions caught 2, 5 and 20 recursive calls above the throw, through direct calls and through a callback, and a backtracking parser that throws on a failed alternative | | `thread-slots`, `thread-slots-cache` | cpu | worker threads reading one published type graph from an `InternTable` and keeping per-thread state for each type they visit: a `ThreadSlots` family, or a direct-mapped cache by id in front of a `Map` by id (the `-cache` entry); under Node.js both are per thread | diff --git a/benchmarks/runtime/workloads/string-iteration.json b/benchmarks/runtime/workloads/string-iteration.json new file mode 100644 index 000000000..21456bd23 --- /dev/null +++ b/benchmarks/runtime/workloads/string-iteration.json @@ -0,0 +1,6 @@ +{ + "entry": "workloads/string-iteration.ts", + "args": [ + "1" + ] +} diff --git a/benchmarks/runtime/workloads/string-iteration.ts b/benchmarks/runtime/workloads/string-iteration.ts new file mode 100644 index 000000000..2abebd635 --- /dev/null +++ b/benchmarks/runtime/workloads/string-iteration.ts @@ -0,0 +1,59 @@ +// Code point iteration over mixed text: a word tokenizer over for-of, a +// character histogram, reversal through spread, and Array.from over +// words. The text mixes ASCII, Latin-1 letters, CJK and emoji. +let seed = 11; +function random(): number { + seed = (Math.imul(seed, 1103515245) + 12345) >>> 0; + return seed / 4294967296; +} +const pieces = [ + "the", "quick", "brown", "fox", "jumps", "over", "lazy", "dog", + "café", "naïve", "über", "señor", "straße", "déjà", "vu", + "日本", "語", "テキスト", "中文", "한국어", + "😀", "🚀", "👍", "🎉", +]; +const punctuation = [" ", " ", " ", ", ", ". ", "! ", "\n"]; +const scale = Number(process.argv[2] ?? "1"); +const lines: string[] = []; +for (let i = 0; i < Math.floor(4000 * scale); i++) { + let line = ""; + const words = 6 + Math.floor(random() * 10); + for (let w = 0; w < words; w++) { + line += pieces[Math.floor(random() * pieces.length)]!; + line += punctuation[Math.floor(random() * punctuation.length)]!; + } + lines.push(line); +} + +function isWordChar(c: string): boolean { + const code = c.codePointAt(0)!; + return (code >= 0x61 && code <= 0x7a) || (code >= 0x41 && code <= 0x5a) || code >= 0xc0; +} + +let tokens = 0; +let tokenUnits = 0; +let reversedLength = 0; +let fromLength = 0; +const histogram = new Map(); +for (let round = 0; round < 30; round++) { + for (const line of lines) { + let word = ""; + for (const c of line) { + if (isWordChar(c)) { + word += c; + continue; + } + if (word !== "") { + tokens++; + tokenUnits += word.length; + if (round === 0) histogram.set(word, (histogram.get(word) ?? 0) + 1); + fromLength += Array.from(word).length; + word = ""; + } + } + reversedLength += [...line].reverse().join("").length; + } +} +const top = [...histogram].sort((a, b) => b[1] - a[1] || (a[0] < b[0] ? -1 : 1)).slice(0, 5); +console.log(top.map(([word, count]) => `${word}:${count}`).join(" ")); +console.log("tokens", tokens, "units", tokenUnits, "reversed", reversedLength, "from", fromLength); diff --git a/docs/content/docs/dependencies.mdx b/docs/content/docs/dependencies.mdx index a9c5ae138..ec1eb0601 100644 --- a/docs/content/docs/dependencies.mdx +++ b/docs/content/docs/dependencies.mdx @@ -141,6 +141,39 @@ CommonJS packages compile with Node.js's module semantics. Each `require()` eval A JavaScript package can contain operations scriptc can't compile, often in functions your program never calls. These don't fail the build: each one compiles to code that throws a runtime error if it ever runs. Coverage reports list them as deferred sites, so review the list and test the parts of the package your program uses. Sites in a module's top-level code are listed under `runs at startup`: that code runs when the module loads, at startup for a module the program imports. +## Fix a package's types + +scriptc trusts declared types: a package's `.d.ts` files, its `@types/` package, and the JSDoc types in its JavaScript. A value that contradicts its declared type throws a `TypeError` instead of flowing on, so a package whose declarations are wrong can stop a program that runs under Node.js. A common case is a member declared `string` or `number` that is sometimes `undefined`. + +Fix such a declaration in your project with a type patch in `package.json`: + +```json:package.json +{ + "scriptc": { + "typePatches": { + "commander@^15.0.0": { + "declarations": { + "HelpContext.helpWidth": "| undefined" + } + } + } + } +} +``` + +Each key names a package and a semver range of the installed versions the patch describes. `declarations` maps a declaration to a type: + +
+
`"| undefined"`
+
A type that starts with `|` widens the declared type, here to `number | undefined`.
+
`"string | null"`
+
Any other type replaces the declared type.
+
+ +A declaration is named by its path: `Options.name` for a member of a type, `parse()` for a function's return type, `parse(text)` for a parameter, `list()[]` for an array's elements, and `Env[string]` for an index signature. `export=()` names the return type of the function the module exports, `default` its default export, and a file prefix such as `index.js:parse()` names a declaration inside one of the package's files. One patch applies to the package's own declarations, its JSDoc, and its `@types/` package. The patched type then holds everywhere, so your code has to handle the `undefined` that a widened declaration allows. An optional `"note"` string can link the upstream issue. + +Patches change only declarations, never the package's code, and values are still checked against the patched types. A build warns when a patch doesn't apply as written: when no range covers the installed version, when a declaration path matches nothing, when the declaration already has the type, or when a path is ambiguous. Report wrong declarations to the package too, so the patch can be removed later. + ## Provenance sources (experimental) `--provenance-sources` compiles a package from the source at the commit its npm provenance attestation names, instead of from its published JavaScript. This lets packages written in TypeScript compile from their original TypeScript. Packages without a usable attestation compile from their installed files as usual. diff --git a/docs/content/limitations/builtin-apis/crypto.mdx b/docs/content/limitations/builtin-apis/crypto.mdx index cba219788..e314fc386 100644 --- a/docs/content/limitations/builtin-apis/crypto.mdx +++ b/docs/content/limitations/builtin-apis/crypto.mdx @@ -31,22 +31,25 @@ `checkPrime`, `checkPrimeSync`, `generatePrime`, and `generatePrimeSync` support bigint, ArrayBuffer, typed-array, Buffer, and DataView values and the `checks`, `safe`, `add`, `rem`, and `bigint` options on native targets, with Node's errors; the callback forms compute before they return and call back in a later macrotask. Candidates and sizes longer than 262144 bits throw a coded SC2020 error, and wasm32-wasi builds are refused. Node calls their callbacks with `undefined` as the error, which a callback whose error parameter is typed `Error | null`, as @types/node declares it, receives as `null`. - RSA, EC, Ed25519, and X25519 private and public KeyObjects come from `generateKeyPairSync`, `generateKeyPair`, `createPrivateKey`, and `createPublicKey`, on native targets. + RSA, RSA-PSS, EC, Ed25519, and X25519 private and public KeyObjects come from `generateKeyPairSync`, `generateKeyPair`, `createPrivateKey`, and `createPublicKey`, on native targets. `generateKeyPairSync` and `generateKeyPair` check their arguments in Node's order, with Node's errors, including the OpenSSL errors for RSA modulus lengths below 512 or above 2147483647 bits and for public exponents of 0, 1, or an even number, which `generateKeyPair` passes to its callback. - RSA keys generate with even modulus lengths from 1024 to 8192 bits and public exponents up to 2147483647; other lengths and exponents that Node accepts, and `paramEncoding: 'explicit'`, throw a coded SC2020 error. + RSA keys generate with modulus lengths from 512 to 8192 bits, an odd length from 2048 bits yielding one bit less as in Node, and odd public exponents up to 4294967295; longer moduli that Node accepts, and `paramEncoding: 'explicit'`, throw a coded SC2020 error. The EC curves are prime256v1 (P-256), secp384r1 (P-384), secp521r1 (P-521), and secp256k1. Keys read and write PEM and DER as PKCS#1, SEC1, PKCS#8, and SubjectPublicKeyInfo, and JWK (Ed25519 and X25519 keys use PKCS#8, SubjectPublicKeyInfo, and JWK); `createPublicKey` also takes a private key or an X.509 certificate. EC, Ed25519, and X25519 keys also read and write the `raw-public` and `raw-private` formats, including compressed EC points. Raw X448, Ed448, and post-quantum keys, EC scalars of 0 or at least the curve order, and the EC point at infinity, which Node accepts, throw a coded SC2020 error. They support `type`, `asymmetricKeyType`, `asymmetricKeyDetails`, `export()`, `equals()`, and `instanceof KeyObject`. `crypto.sign`, `crypto.verify`, `createSign`, and `createVerify` support RSA PKCS#1 v1.5 and PSS padding and ECDSA with DER and IEEE P1363 encodings over SHA-1 and SHA-2 digests, and Ed25519 through `crypto.sign` and `crypto.verify` with a `null` or `undefined` algorithm. + RSA-PSS keys follow their RSASSA-PSS-params as Node does: PSS padding only, the key's digest and MGF1 digest, and salts of at least the key's length, with OpenSSL's errors; their details name the parameters, they export as PKCS#8 and SubjectPublicKeyInfo only, and digests other than SHA-1 and SHA-2 in their options throw a coded SC2020 error. Ed25519 signing, verification, and private keys link when a string literal in the program names `ed25519` (in any case); without one they throw a coded SC2020 error, while Ed25519 public keys still import and export. X509Certificate supports `publicKey`, `verify()`, `checkIssued()`, `checkPrivateKey()`, `checkHost()`, `checkEmail()`, `checkIP()`, `toLegacyObject()`, `toString()`, and `toJSON()`, and `String()`, template literals, and `+` give its PEM text as `toString()` does. `publicKey`, `verify()`, and `checkPrivateKey()` are refused in wasm32-wasi builds, and an Ed25519 certificate's `verify()` needs the `ed25519` literal like the other Ed25519 operations. Own-key walks over a certificate (`Object.keys()`, `Object.entries()`, spread, `for...in`, and `hasOwnProperty`) are refused, since Node keeps its fields as prototype getters. `JSON.stringify()` and `console.log()` of a certificate, a certificate stored in an `unknown` or `any` slot, and its methods on a union-typed value are also refused. Sign and Verify values accumulate `update()` data and are not writable streams. - Encrypted private keys (`passphrase`, `cipher`), the other key types (`rsa-pss`, `dsa`, `ed448`, `x448`, and the rest), and wasm32-wasi builds throw a coded SC2020 error or are refused. + Private keys export encrypted (`cipher`, `passphrase`) with aes-128, aes-192, and aes-256 in CBC and ECB modes and with des-ede3-cbc, PKCS#8 as PBES2 with PBKDF2-HMAC-SHA256 and PKCS#1 and SEC1 as traditional PEM, and the ciphers OpenSSL refuses for a form throw its errors; a cipher option can also be an OID; other ciphers Node encrypts with throw a coded SC2020 error. + Encrypted private keys import with a `passphrase` from PBES2 with HMAC-SHA PRFs and the CBC and ECB ciphers, and from traditional PEM with the CBC ciphers, with Node's errors for a missing or wrong passphrase; other schemes and ciphers throw a coded SC2020 error once a passphrase is given. + The other key types (`dsa`, `ed448`, `x448`, and the rest) and wasm32-wasi builds throw a coded SC2020 error or are refused. Comparing keys of different types with `equals()` answers false; Node also leaves an OpenSSL error queued, which a later key operation can report. `publicEncrypt`/`privateDecrypt` remain unsupported. The WebCrypto `crypto` global, which node:crypto also exports as `webcrypto`, supports `randomUUID()` and `getRandomValues()`, also read as values from untyped JavaScript. diff --git a/packages/cli-darwin-arm64/package.json b/packages/cli-darwin-arm64/package.json index bac01cc44..1a8ee5d2c 100644 --- a/packages/cli-darwin-arm64/package.json +++ b/packages/cli-darwin-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-darwin-arm64", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for darwin-arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-darwin-x64/package.json b/packages/cli-darwin-x64/package.json index 6c2fb3123..0cb24b908 100644 --- a/packages/cli-darwin-x64/package.json +++ b/packages/cli-darwin-x64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-darwin-x64", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for darwin-x64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-linux-arm64-gnu/package.json b/packages/cli-linux-arm64-gnu/package.json index 289d8d6bd..3a1c7488d 100644 --- a/packages/cli-linux-arm64-gnu/package.json +++ b/packages/cli-linux-arm64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-linux-arm64-gnu", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for linux-arm64-gnu", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-linux-arm64-musl/package.json b/packages/cli-linux-arm64-musl/package.json index 3a601ca33..e671e926b 100644 --- a/packages/cli-linux-arm64-musl/package.json +++ b/packages/cli-linux-arm64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-linux-arm64-musl", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for linux-arm64-musl", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-linux-x64-gnu/package.json b/packages/cli-linux-x64-gnu/package.json index 8f6b1aa5e..2b56308fd 100644 --- a/packages/cli-linux-x64-gnu/package.json +++ b/packages/cli-linux-x64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-linux-x64-gnu", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for linux-x64-gnu", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-linux-x64-musl/package.json b/packages/cli-linux-x64-musl/package.json index ff0d1d686..4ca4c2837 100644 --- a/packages/cli-linux-x64-musl/package.json +++ b/packages/cli-linux-x64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-linux-x64-musl", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for linux-x64-musl", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli-win32-x64-msvc/package.json b/packages/cli-win32-x64-msvc/package.json index e94ba31df..d734c0807 100644 --- a/packages/cli-win32-x64-msvc/package.json +++ b/packages/cli-win32-x64-msvc/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/cli-win32-x64-msvc", - "version": "0.4.4", + "version": "0.4.5", "description": "Native scriptc compiler for win32-x64-msvc", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli/package.json b/packages/cli/package.json index 26cf10f07..59f9db90a 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,6 +1,6 @@ { "name": "scriptc", - "version": "0.4.4", + "version": "0.4.5", "description": "Compile ordinary TypeScript and JavaScript to small, fast native executables and WebAssembly modules that run without Node.js", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/cli/test/native-link-info.test.ts b/packages/cli/test/native-link-info.test.ts index ef07c838d..f63cc790c 100644 --- a/packages/cli/test/native-link-info.test.ts +++ b/packages/cli/test/native-link-info.test.ts @@ -72,7 +72,7 @@ describe.runIf(supported)("macOS arm64 native link info", () => { minimum_os: "14.0", }, program: { object, entry_symbol: "main" }, - runtime_abi: { version: 44, marker: "scr_runtime_abi_v44" }, + runtime_abi: { version: 45, marker: "scr_runtime_abi_v45" }, runtime_pack: { kind: "precompiled", package: "@scriptc/runtime-darwin-arm64", @@ -165,6 +165,6 @@ describe.runIf(supported)("macOS arm64 native link info", () => { () => null, (failure: { stderr?: string }) => failure, ); - expect(error?.stderr).toContain("scr_runtime_abi_v44"); + expect(error?.stderr).toContain("scr_runtime_abi_v45"); }); }); diff --git a/packages/compiler/ambient/scriptc-node-fallback.d.ts b/packages/compiler/ambient/scriptc-node-fallback.d.ts index 1924baece..058b262ea 100644 --- a/packages/compiler/ambient/scriptc-node-fallback.d.ts +++ b/packages/compiler/ambient/scriptc-node-fallback.d.ts @@ -2914,6 +2914,8 @@ declare module "crypto" { export interface AsymmetricKeyExportOptions { type: "pkcs1" | "spki" | "pkcs8" | "sec1"; format: T; + cipher?: string | undefined; + passphrase?: string | Buffer | undefined; } export interface AsymmetricKeyDetails { modulusLength?: number; @@ -2938,6 +2940,7 @@ declare module "crypto" { format?: "pem" | "der" | undefined; type?: "pkcs1" | "pkcs8" | "sec1" | undefined; encoding?: string | undefined; + passphrase?: string | Buffer | undefined; } export interface PublicKeyInput { key: string | Buffer; @@ -2974,12 +2977,22 @@ declare module "crypto" { modulusLength: number; publicExponent?: number | undefined; publicKeyEncoding: { type: "pkcs1" | "spki"; format: PubF }; - privateKeyEncoding: { type: "pkcs1" | "pkcs8"; format: PrivF }; + privateKeyEncoding: { + type: "pkcs1" | "pkcs8"; + format: PrivF; + cipher?: string | undefined; + passphrase?: string | undefined; + }; } export interface ECKeyPairOptions { namedCurve: string; publicKeyEncoding: { type: "pkcs1" | "spki"; format: PubF }; - privateKeyEncoding: { type: "sec1" | "pkcs8"; format: PrivF }; + privateKeyEncoding: { + type: "sec1" | "pkcs8"; + format: PrivF; + cipher?: string | undefined; + passphrase?: string | undefined; + }; } export function generateKeyPairSync( type: "rsa", @@ -3007,7 +3020,12 @@ declare module "crypto" { ): KeyPairKeyObjectResult; export interface ED25519KeyPairOptions { publicKeyEncoding: { type: "spki"; format: PubF }; - privateKeyEncoding: { type: "pkcs8"; format: PrivF }; + privateKeyEncoding: { + type: "pkcs8"; + format: PrivF; + cipher?: string | undefined; + passphrase?: string | undefined; + }; } export function generateKeyPairSync( type: "ed25519", @@ -3020,7 +3038,12 @@ declare module "crypto" { export function generateKeyPairSync(type: "ed25519", options?: {}): KeyPairKeyObjectResult; export interface X25519KeyPairOptions { publicKeyEncoding: { type: "spki"; format: PubF }; - privateKeyEncoding: { type: "pkcs8"; format: PrivF }; + privateKeyEncoding: { + type: "pkcs8"; + format: PrivF; + cipher?: string | undefined; + passphrase?: string | undefined; + }; } export function generateKeyPairSync( type: "x25519", @@ -5322,10 +5345,10 @@ declare module "node:zlib" { * with the real port discoverable as address().port (the composed read * is the lowered form — bare address() has no lowering); close([cb]) * firing once connections drain; socket write/end/destroy/pipe and the - * data/end/close/error/connect events (on and once — listeners take at - * most the declared parameter). write/end return void here (Node returns - * the backpressure boolean / the socket); on/once return void (chaining - * is fenced). connect's host is a numeric IP or "localhost" (pinned to + * data/end/close/error/connect/drain events (on and once — listeners + * take at most the declared parameter). write returns Node's + * backpressure boolean; end returns void here (Node returns the socket); + * on/once return void (chaining is fenced). connect's host is a numeric IP or "localhost" (pinned to * 127.0.0.1 — SEMANTICS.md); every failure is the async 'error' event, * and an 'error' with no listener exits 1 like an unhandled EventEmitter * 'error'. */ @@ -5357,7 +5380,13 @@ declare module "net" { readonly bytesWritten: number; /* write's callback runs once the data is flushed, with null; end's * once the socket finishes. */ - write(data: string | Uint8Array, callback?: (err?: Error | null) => void): void; + /* false when the bytes this write left unsent reach the 65536-byte + * high-water mark, as in Node; 'drain' fires when they are all sent. */ + write(data: string | Uint8Array, callback?: (err?: Error | null) => void): boolean; + /* the bytes written and not yet sent */ + readonly writableLength: number; + /* a write answered false and 'drain' has not fired yet */ + readonly writableNeedDrain: boolean; end(callback?: () => void): void; end(data: string | Uint8Array, callback?: () => void): void; destroy(): void; @@ -5405,6 +5434,7 @@ declare module "net" { | "connect" | "ready" | "timeout" + | "drain" | "readable" | "finish" | "secureConnect" @@ -5429,6 +5459,7 @@ declare module "net" { | "connect" | "ready" | "timeout" + | "drain" | "readable" | "finish" | "secureConnect" @@ -5452,6 +5483,7 @@ declare module "net" { | "connect" | "ready" | "timeout" + | "drain" | "readable" | "finish" | "secureConnect" diff --git a/packages/compiler/package.json b/packages/compiler/package.json index 570c0a1a5..03af418a5 100644 --- a/packages/compiler/package.json +++ b/packages/compiler/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/compiler", - "version": "0.4.4", + "version": "0.4.5", "description": "The scriptc compiler — TypeScript/JavaScript frontend, typed IR, LLVM backend", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/compiler/src/backend/llvm/bytes-alias.ts b/packages/compiler/src/backend/llvm/bytes-alias.ts index b7a672754..2dff9f70b 100644 --- a/packages/compiler/src/backend/llvm/bytes-alias.ts +++ b/packages/compiler/src/backend/llvm/bytes-alias.ts @@ -9,15 +9,25 @@ import type { LlvmEmitterContext } from "./expr-context.js"; * receiver slots loop-invariant. Untagged accesses and calls still alias * every category. * - * Header loads are also `!invariant.load`: no store or call between the - * construction of a view and its release changes them. Without it, a data - * pointer load ahead of a bounds branch is sunk into the in-bounds block, - * and LICM keeps a conditionally executed load inside the loop. */ + * Header loads that only bound or address an access to the view are also + * `!invariant.load`: no store or call between the construction of a view + * and its release changes them, and their values reach only that access, + * which runs while the view is alive. Without it, a data pointer load + * ahead of a bounds branch is sunk into the in-bounds block, and LICM keeps + * a conditionally executed load inside the loop. */ export function bytesHeaderAlias(host: LlvmEmitterContext): string { const tag = host.memoryAliasAttachment("bytes:header"); return tag === "" ? "" : `${tag}, !invariant.load !{}`; } +/** A header load whose value the program reads (`length`, `byteLength`) + * keeps the alias tag only. The value can outlive the view, and LLVM may + * move an invariant load to its use, past the release that frees the + * view. */ +export function bytesHeaderValueAlias(host: LlvmEmitterContext): string { + return host.memoryAliasAttachment("bytes:header"); +} + /** Element storage's tag, and the alias scope of the access being emitted * in a loop version's fast copy (bytesAccessScope). */ export function bytesElementAlias(host: LlvmEmitterContext): string { diff --git a/packages/compiler/src/backend/llvm/dense-array-access.ts b/packages/compiler/src/backend/llvm/dense-array-access.ts index df37d9c58..e002ecd8a 100644 --- a/packages/compiler/src/backend/llvm/dense-array-access.ts +++ b/packages/compiler/src/backend/llvm/dense-array-access.ts @@ -453,7 +453,11 @@ export function emitDenseArrayRead( B.br(join); B.startBlock(slow); const slowValue = B.tmp(); - const attrs = acc === "ref" && retain ? " cold" : READ_FALLBACK; + // ToNumber of a foreign element (SCR_ARR_FOREIGN) can run user code. + const attrs = + (acc === "ref" && retain) || (getter === "scr_arr_get_number" && host.mayHoldForeign(element)) + ? " cold" + : READ_FALLBACK; B.line(`${slowValue} = call ${ty} @${getter}(ptr ${array}, double ${index.name})${attrs}`); let slowInt: string | null = null; if (fastInt !== null) { @@ -610,6 +614,28 @@ export function emitDenseArraySet( B.line(`store ptr ${value}, ptr ${valuePtr}${host.fieldAliasAttachment(valuePtr)}`); } }; + if ((acc === "f64" || kind === "ref") && host.mayHoldForeign(element)) { + // Past the hole-free prefix a slot may hold a foreign element, whose + // value the runtime setter releases; below it every slot is a value. + B.br(slow); + B.startBlock(filledLabel); + const filledValuePtr = valuePointer(host, guard, kind); + let filledOld: string | null = null; + if (kind === "ref") { + filledOld = B.tmp(); + B.line( + `${filledOld} = load ptr, ptr ${filledValuePtr}${host.fieldAliasAttachment(filledValuePtr)}`, + ); + } + storeValue(filledValuePtr); + if (filledOld) host.releaseHeap(filledOld, element); + B.br(join); + B.startBlock(slow); + call(" cold"); + B.br(join); + B.startBlock(join); + return; + } const stated = B.newLabel("arr.set.states"), plain = B.newLabel("arr.set.plain"), plainStore = B.newLabel("arr.set.values"), diff --git a/packages/compiler/src/backend/llvm/dyn.ts b/packages/compiler/src/backend/llvm/dyn.ts index 9ba8c3c87..12ea99688 100644 --- a/packages/compiler/src/backend/llvm/dyn.ts +++ b/packages/compiler/src/backend/llvm/dyn.ts @@ -1980,6 +1980,87 @@ export class LlDyn { /** `sc_dc_(ptr d, ptr path) -> T` — validate the checked-dynamic tree against T and * BUILD the typed value (+1), or throw the catchable path-annotated * TypeError and return a dummy with the pending flag set. */ + /** The array a live view's snapshot commits back into, for element types + * whose arrays can hold foreign elements (LlEmitter.mayHoldForeign): + * like dynCheckHelper's array case on a checked array `%d`, except that + * an element the element type cannot hold is stored as a foreign element + * (undefined as a present undefined) instead of failing the conversion. + * Boolean arrays, whose slots cannot hold one, keep the failing check. */ + dynViewArrayHelper(t: IrType & { kind: "array" }): string { + const builderKey = `view:${typeKey(t)}`; + const existing = this.dynBuilders.get(builderKey); + if (existing) return existing; + const name = `sc_dva_${this.dynBuilders.size}`; + this.dynBuilders.set(builderKey, name); + const host = this.host; + const elem = t.elem; + const check = this.dynCheckHelper(elem); + const match = this.dynMatchHelper(elem); + const B = new BlockBuilder(); + const n = this.lenOf(B, "%d"); + const a = B.tmp(); + B.line(`${a} = ${arrNewCall(host, elem, n)}`); + const items = this.itemsOf(B, "%d"); + const acc = elemAccess(elem); + const accTy = acc === "f64" ? "double" : acc === "bool" ? "i1" : "ptr"; + host.declare( + `declare double @scr_arr_push_${acc}(ptr, ${acc === "bool" ? "i1 zeroext" : accTy})`, + ); + host.declare(`declare zeroext i1 @scr_dyn_arr_has_index(ptr, ${this.S})`); + host.declare(`declare void @scr_arr_set_len(ptr, double)`); + host.declare(`declare zeroext i1 @scr_arr_set_foreign(ptr, double, ptr)`); + this.i64Loop(B, "dva", n, (i) => { + const index = B.tmp(); + B.line(`${index} = ${sizeToDouble(this.S)} ${this.S} ${i} to double`); + const present = B.tmp(); + B.line(`${present} = call zeroext i1 @scr_dyn_arr_has_index(ptr %d, ${this.S} ${i})`); + const valueLabel = B.newLabel("dva.present"); + const holeLabel = B.newLabel("dva.hole"); + const typedLabel = B.newLabel("dva.typed"); + const foreignLabel = B.newLabel("dva.foreign"); + const doneLabel = B.newLabel("dva.done"); + B.condBr(present, valueLabel, holeLabel); + B.startBlock(holeLabel); + const length = B.tmp(); + B.line(`${length} = fadd double ${index}, 1.0`); + B.line(`call void @scr_arr_set_len(ptr ${a}, double ${length})`); + B.br(doneLabel); + B.startBlock(valueLabel); + const e = this.itemAt(B, items, i); + const fits = B.tmp(); + B.line(`${fits} = call zeroext i1 @${match}(ptr ${e})`); + B.condBr(fits, typedLabel, foreignLabel); + B.startBlock(foreignLabel); + const kept = B.tmp(); + B.line(`${kept} = call zeroext i1 @scr_arr_set_foreign(ptr ${a}, double ${index}, ptr ${e})`); + B.condBr(kept, doneLabel, typedLabel); + B.startBlock(typedLabel); + const v = B.tmp(); + B.line(`${v} = call ${this.valTy(elem)} @${check}(ptr ${e}, ptr null)`); + this.pendingBail( + B, + "dva", + () => { + host.declare(`declare void @scr_arr_release(ptr)`); + B.line(`call void @scr_arr_release(ptr ${a})`); + }, + "ptr null", + ); + const pushed = B.tmp(); + B.line(`${pushed} = call double @scr_arr_push_${acc}(ptr ${a}, ${accTy} ${v})`); + B.br(doneLabel); + B.startBlock(doneLabel); + }); + B.terminate(`ret ptr ${a}`); + this.defs.push( + `define internal ptr @${name}(ptr %d, ptr %path) ${FN_ATTRS} { ; view commit ${typeKey(t)}`, + B.render(), + `}`, + ``, + ); + return name; + } + dynCheckHelper(t: IrType, preserveRefs = false): string { // A `(T | undefined)[]` stored as T elements also takes undefined ones. const key = checkedConversionKey(t); @@ -3686,7 +3767,25 @@ export class LlDyn { B.br(doneLabel); B.startBlock(presentLabel); B.line(`${undefinedState} = fcmp oeq double ${state}, 2.0`); // SCR_ARR_UNDEFINED - B.terminate(`br i1 ${undefinedState}, label %${undefinedLabel}, label %${valueLabel}`); + if (host.mayHoldForeign(t.elem)) { + // A foreign element (SCR_ARR_FOREIGN) holds its own value. + const typedLabel = B.newLabel("tda.typed"); + const foreignLabel = B.newLabel("tda.foreign"); + B.terminate(`br i1 ${undefinedState}, label %${undefinedLabel}, label %${typedLabel}`); + B.startBlock(typedLabel); + const foreign = B.tmp(); + B.line(`${foreign} = fcmp oeq double ${state}, 3.0`); + B.condBr(foreign, foreignLabel, valueLabel); + B.startBlock(foreignLabel); + host.declare(`declare ptr @scr_arr_get_foreign(ptr, double)`); + host.declare(`declare void @scr_dyn_arr_push(ptr, ptr)`); + const payload = B.tmp(); + B.line(`${payload} = call ptr @scr_arr_get_foreign(ptr %v, double ${i}) ; +1`); + B.line(`call void @scr_dyn_arr_push(ptr ${d}, ptr ${payload})`); + B.br(doneLabel); + } else { + B.terminate(`br i1 ${undefinedState}, label %${undefinedLabel}, label %${valueLabel}`); + } B.startBlock(undefinedLabel); const retained = this.retainDyn(B, this.undef(B)); B.line(`call void @scr_dyn_arr_push(ptr ${d}, ptr ${retained})`); diff --git a/packages/compiler/src/backend/llvm/emit-module.ts b/packages/compiler/src/backend/llvm/emit-module.ts index 5ed2658e7..33bae2043 100644 --- a/packages/compiler/src/backend/llvm/emit-module.ts +++ b/packages/compiler/src/backend/llvm/emit-module.ts @@ -618,6 +618,23 @@ export function emitModuleParts(emitter: LlEmitter): string[] { regexMatchers.set(key, code); for (const decl of code.decls) emitter.declare(decl); } + // Replacers of global substitutions with a literal replacement; the call + // sites register only patterns inside the compiled subset. + const regexReplacers: RegexMatcherCode[] = []; + for (const rr of emitter.regexReplacers.values()) { + const code = compileRegexMatcher( + rr.sym, + rr.pattern, + rr.flags, + emitter.sizeType, + !emitter.wasi, + rr.replacement, + ); + if (code === null) + throw new InternalCompilerError(`llvm emitter bug: replacer ${rr.sym} did not compile`); + regexReplacers.push(code); + for (const decl of code.decls) emitter.declare(decl); + } const out: string[] = [ `; Generated by scriptc (LLVM backend) from ${emitter.mod.sourceFile}. Do not edit.`, @@ -901,6 +918,7 @@ export function emitModuleParts(emitter: LlEmitter): string[] { } if (emitter.regexInstances.size > 0) out.push(``); for (const code of regexMatchers.values()) out.push(...code.lines, ``); + for (const code of regexReplacers) out.push(...code.lines, ``); for (const [, inst] of emitter.templateStringsInstances) { // One immortal ScrArr per tagged-template site: a data global of // interned cooked-string literals, and the ScrArr header over it diff --git a/packages/compiler/src/backend/llvm/emit-statements.ts b/packages/compiler/src/backend/llvm/emit-statements.ts index dc4e5090c..a43cf3534 100644 --- a/packages/compiler/src/backend/llvm/emit-statements.ts +++ b/packages/compiler/src/backend/llvm/emit-statements.ts @@ -91,6 +91,7 @@ import { } from "./string-construction.js"; import { emitStringInputs } from "./string-lifetimes.js"; import { LOOPS } from "./loop-state.js"; +import { loopAccessesTypedArrays } from "../loop-polls.js"; /** A callback loop version copies the loop; keep the copies small. */ const MAX_CALLBACK_VERSION_COST = 400; @@ -1211,9 +1212,68 @@ export function emitStmtBody(emitter: LlEmitter, s: IrStmt): void { B.line(`store ptr ${fresh}, ptr ${slot}`); } }; - const workerBudget = emitter.workerLoopBudget(s); + // A counted loop over typed arrays, or a loop to a typed array's + // length, polls between blocks of WORKER_POLL_BLOCK iterations instead + // of counting a budget in its body: the inner loop over a block then + // has no exit besides its own bound, which LLVM can vectorize. A typed + // array's length is fixed, so each block can take it as its limit when + // the loop does not assign the array's binding. + const stripMined = + capturedLets.length === 0 && + integerSlot !== null && + ((countedLoop !== null && + countedLimit !== null && + Math.abs(countedLoop.step) * WORKER_POLL_BLOCK <= 2 ** 52) || + (integerLoop !== null && integerArrayLoop === null && integerLoop.stableReceiver)) && + emitter.workerLoopPolls(s) && + loopAccessesTypedArrays(s); + const workerBudget = stripMined ? null : emitter.workerLoopBudget(s); + // One slot for every copy of the loop: a speculative copy resumes an + // iteration in another copy's body, which then reads this bound. + let blockBound: string | null = null; + if (stripMined) { + blockBound = B.slot(); + B.entryAllocas.push( + `${blockBound} = alloca ${countedLoop ? "i64" : emitter.sizeType} ; cancellation poll block bound`, + ); + } freshenBindings(); emitter.emitLoopCopies(s, capturedLets.length === 0, le, (exit) => { + if (stripMined) { + const counter: StripMinedCounter = countedLoop + ? { + slot: integerSlot!, + type: "i64", + signed: true, + step: countedLoop.step, + followers: countedLoop.followers.map((f, i) => ({ + slot: followerSlots[i]!, + step: f.step, + })), + limit: () => countedLimit!, + } + : { + slot: integerSlot!, + type: emitter.sizeType, + signed: false, + step: 1, + followers: [], + limit: () => { + const receiver = emitter.emitStableReceiver(integerLoop!.limitReceiver, []); + const lenPtr = B.tmp(); + const len = B.tmp(); + B.line( + `${lenPtr} = getelementptr inbounds %ScrBytes, ptr ${receiver.name}, i64 0, i32 1`, + ); + B.line( + `${len} = load ${emitter.sizeType}, ptr ${lenPtr}${bytesHeaderAlias(emitter)}`, + ); + return len; + }, + }; + emitStripMinedCountedLoop(emitter, s, counter, blockBound!, exit); + return; + } const lc = B.newLabel("loop.c"); const lb = B.newLabel("loop.b"); const lu = @@ -2565,6 +2625,102 @@ function emitInt32KindLoop( /** At the top of a loop copy's header: lets a speculative loop find its * ordinary copy's header, and keeps its third copy out of the loop. */ +/** Iterations between cancellation polls of a strip-mined counted loop. */ +const WORKER_POLL_BLOCK = 64; + +/** The counter of a strip-mined loop: its slot, the LLVM integer type and + * signedness of its values, its step, the counters stepped beside it, and + * the limit, emitted in the outer header before each block. */ +interface StripMinedCounter { + slot: string; + type: string; + signed: boolean; + step: number; + followers: { slot: string; step: number }[]; + limit: () => string; +} + +/** A counted `for` loop of a worker program as blocks of WORKER_POLL_BLOCK + * iterations, with the cancellation poll between blocks: the outer header + * tests the counter against the loop's limit, each block runs the body up + * to the nearer of the limit and WORKER_POLL_BLOCK steps on, and the poll + * follows the block. It polls as often as the budget it replaces, and the + * inner loop's only exit besides `break` and `return` is its bound. */ +function emitStripMinedCountedLoop( + emitter: LlEmitter, + s: Extract, + counter: StripMinedCounter, + boundSlot: string, + exit: string, +): void { + const B = emitter.B; + const { slot, type, signed, step } = counter; + const ascending = step > 0; + const compare = !signed ? "ult" : ascending ? "slt" : "sgt"; + const saturating = `llvm.${signed ? "sadd" : "uadd"}.sat.${type}`; + const nearer = `llvm.${!signed ? "umin" : ascending ? "smin" : "smax"}.${type}`; + const wrap = signed ? "nsw" : "nuw"; + const lc = B.newLabel("loop.c"); + const block = B.newLabel("loop.block"); + const bc = B.newLabel("loop.bc"); + const lb = B.newLabel("loop.b"); + const lu = B.newLabel("loop.u"); + const poll = B.newLabel("loop.poll"); + emitter.declare(`declare ${type} @${saturating}(${type}, ${type})`); + emitter.declare(`declare ${type} @${nearer}(${type}, ${type})`); + B.br(lc); + B.startBlock(lc); + enterLoopHeader(emitter, lc); + const start = B.tmp(); + const open = B.tmp(); + B.line(`${start} = load ${type}, ptr ${slot}`); + const limit = counter.limit(); + B.line(`${open} = icmp ${compare} ${type} ${start}, ${limit}`); + B.condBr(open, block, exit); + B.startBlock(block); + const stepped = B.tmp(); + const bound = B.tmp(); + B.line( + `${stepped} = call ${type} @${saturating}(${type} ${start}, ${type} ${step * WORKER_POLL_BLOCK})`, + ); + B.line(`${bound} = call ${type} @${nearer}(${type} ${stepped}, ${type} ${limit})`); + B.line(`store ${type} ${bound}, ptr ${boundSlot}`); + B.br(bc); + B.startBlock(bc); + const index = B.tmp(); + const end = B.tmp(); + const inBlock = B.tmp(); + B.line(`${index} = load ${type}, ptr ${slot}`); + B.line(`${end} = load ${type}, ptr ${boundSlot}`); + B.line(`${inBlock} = icmp ${compare} ${type} ${index}, ${end}`); + B.condBr(inBlock, lb, poll); + B.startBlock(lb); + emitter.jumpTargets.push({ + kind: "loop", + brkLabel: exit, + contLabel: lu, + labels: s.labels, + frameDepth: emitter.frames.length, + scopeDepth: emitter.scopes.length, + finallyDepth: emitter.finallyStack.length, + }); + emitter.emitBlock(s.body); + emitter.jumpTargets.pop(); + B.br(lu); + B.startBlock(lu); + for (const { slot: stepSlot, step: by } of [{ slot, step }, ...counter.followers]) { + const old = B.tmp(); + const next = B.tmp(); + B.line(`${old} = load ${type}, ptr ${stepSlot}`); + B.line(`${next} = add ${wrap} ${type} ${old}, ${by}`); + B.line(`store ${type} ${next}, ptr ${stepSlot}`); + } + B.br(bc); + B.startBlock(poll); + emitter.emitWorkerPoll(); + B.br(lc); +} + function enterLoopHeader(emitter: LlEmitter, header: string): void { const dispatch = emitter.speculationHeader; emitter.speculationHeader = null; diff --git a/packages/compiler/src/backend/llvm/emitter.ts b/packages/compiler/src/backend/llvm/emitter.ts index cb9b417db..56411120a 100644 --- a/packages/compiler/src/backend/llvm/emitter.ts +++ b/packages/compiler/src/backend/llvm/emitter.ts @@ -1307,6 +1307,12 @@ export class LlEmitter { string, { sym: string; src: string; fl: string; pattern: string; flags: string } >(); + /** Build-time replacers: one per (pattern, flags, replacement) of a + * global substitution with a literal replacement (regex-matchers.ts). */ + readonly regexReplacers = new Map< + string, + { sym: string; pattern: string; flags: string; replacement: string } + >(); /** Interned tagged-template strings objects: per-site key → { symbol, * interned cooked-literal refs }. One immortal ScrArr of string slots * per template SITE (the spec's per-occurrence identity — the C @@ -1637,6 +1643,9 @@ export class LlEmitter { * commit back into their objects. Library builds keep their commits: * their values cross a native boundary. */ readonly liveViewsReadOnly: boolean; + /** IrModule.foreignElementKeys: element keys whose arrays can hold + * SCR_ARR_FOREIGN elements. */ + readonly foreignElementKeys: ReadonlySet; /** The program freezes, seals or prevents extensions of untyped values * (IrModule.untypedIntegrity): the commits of class views store the * view's integrity state in the instance's integrity word. */ @@ -1929,6 +1938,7 @@ export class LlEmitter { !this.untypedIntegrity && !dynamicStores.all && dynamicStores.names.size === 0; + this.foreignElementKeys = new Set(mod.foreignElementKeys ?? []); this.int32Abis = this.debug === null ? findInt32Abis(mod, this.int32Slots) : new Map(); this.immortalGlobals = new Set(mod.contracts?.immortalGlobals ?? []); this.programLifetime = new Set(mod.contracts?.programLifetime ?? []); @@ -2224,6 +2234,7 @@ export class LlEmitter { declare: (decl) => this.declare(decl), needOom: () => this.needOom(), needBadTag: () => this.needBadTag(), + mayHoldForeign: (elem) => this.mayHoldForeign(elem), internLiteral: (text) => this.internLiteral(text), cstr: (text) => this.cstr(text), mapOps: (operands) => this.mapOps(operands), @@ -3934,13 +3945,19 @@ export class LlEmitter { } } + /** Whether `loop` needs a cancellation poll: in a worker program, unless + * the loop already runs a pending test on every iteration, which observes + * a terminate request itself (a separate poll would only add work). */ + workerLoopPolls(loop: Extract): boolean { + return ( + this.mod.workers === true && !loopTestsEveryIteration(loop, (e) => this.callTestsPending(e)) + ); + } + workerLoopBudget( loop: Extract, ): string | null { - if (!this.mod.workers) return null; - // A loop that runs a pending test on every iteration observes a - // terminate request through it; a separate poll would only add work. - if (loopTestsEveryIteration(loop, (e) => this.callTestsPending(e))) return null; + if (!this.workerLoopPolls(loop)) return null; const slot = this.B.slot(); this.B.entryAllocas.push(`${slot} = alloca i32 ; worker cancellation budget`); this.B.line(`store i32 0, ptr ${slot}`); @@ -4058,14 +4075,19 @@ export class LlEmitter { B.condBr(due, poll, next); B.startBlock(poll); B.line(`store i32 63, ptr ${slot}`); - // Calls and throwing operations retain their own checks. Amortize the - // extra cancellation poll across bounded loop batches so scalar work - // does not cross the runtime ABI on every iteration. - this.emitPendingCheck(undefined, true); + this.emitWorkerPoll(); B.br(next); B.startBlock(next); } + /** The cancellation poll itself. Calls and throwing operations retain + * their own checks. Loops amortize this extra poll across bounded + * batches of iterations so scalar work does not cross the runtime ABI on + * every iteration. */ + emitWorkerPoll(): void { + this.emitPendingCheck(undefined, true); + } + /** Moves an already-evaluated value into the runtime's exception cell — * the `throw` statement's kind dispatch (stmts.ts's), shared with * every synthetic thrower. Ownership of a refcounted payload must have @@ -5433,6 +5455,8 @@ export class LlEmitter { return addNumbered(this.threadSlotsBlocks, key, value as string, "@sc_slots_", names); case "regex": return addNumberedSymbol(this.regexInstances, key, value as never, "sc_re_", names); + case "regexReplacer": + return addNumberedSymbol(this.regexReplacers, key, value as never, "sc_rr_", names); case "templateStrings": return addNumberedSymbol( this.templateStringsInstances, @@ -6710,6 +6734,12 @@ export class LlEmitter { } } + /** Arrays of this element type can hold SCR_ARR_FOREIGN elements. */ + mayHoldForeign(elem: IrType): boolean { + if (this.foreignElementKeys.size === 0) return false; + return this.foreignElementKeys.has("*") || this.foreignElementKeys.has(typeKey(elem)); + } + /** Lifetime regions in modules with lifetimeRegions (llvm/regions.ts). * Created with the class graph; null elsewhere. */ regions: LifetimeRegions | null = null; diff --git a/packages/compiler/src/backend/llvm/expr-bytes.ts b/packages/compiler/src/backend/llvm/expr-bytes.ts index a50464cd6..6f0b0380d 100644 --- a/packages/compiler/src/backend/llvm/expr-bytes.ts +++ b/packages/compiler/src/backend/llvm/expr-bytes.ts @@ -8,7 +8,7 @@ import { alwaysExactInteger, exactInteger } from "./integer-values.js"; import { byteNumberAccess } from "../../ir/byte-numbers.js"; import { emitByteNumber } from "./byte-numbers.js"; import { integerOrNaNIndex, integerOrNaNRange } from "./wrapping-sums.js"; -import { bytesElementAlias, bytesHeaderAlias } from "./bytes-alias.js"; +import { bytesElementAlias, bytesHeaderAlias, bytesHeaderValueAlias } from "./bytes-alias.js"; const BYTES_NUM_KIND: Record = { u8: { kind: 0, le: false }, @@ -160,12 +160,19 @@ export function emitBytesIndex( return idx; } -/** The element count of a typed array, in the target's size type. */ -export function emitBytesHeaderLength(host: LlvmEmitterContext, receiver: string): string { +/** The element count of a typed array, in the target's size type. `value`: + * the program reads the count itself (bytesHeaderValueAlias), rather than + * only bounding an access. */ +export function emitBytesHeaderLength( + host: LlvmEmitterContext, + receiver: string, + value = false, +): string { const p = host.B.tmp(); const len = host.B.tmp(); + const alias = value ? bytesHeaderValueAlias(host) : bytesHeaderAlias(host); host.B.line(`${p} = getelementptr inbounds %ScrBytes, ptr ${receiver}, i64 0, i32 1`); - host.B.line(`${len} = load ${host.sizeType}, ptr ${p}${bytesHeaderAlias(host)}`); + host.B.line(`${len} = load ${host.sizeType}, ptr ${p}${alias}`); return len; } @@ -184,7 +191,7 @@ export function emitBytesLength( bytes: boolean, ): LlValue { const B = host.B; - const len = emitBytesHeaderLength(host, receiver); + const len = emitBytesHeaderLength(host, receiver, true); const size = BYTES_ELEMENT_SIZE[elem]; const count = bytes && size !== 1 ? B.tmp() : len; if (count !== len) B.line(`${count} = shl ${host.sizeType} ${len}, ${Math.log2(size)}`); diff --git a/packages/compiler/src/backend/llvm/expr-containers.ts b/packages/compiler/src/backend/llvm/expr-containers.ts index 307481737..696dad3ab 100644 --- a/packages/compiler/src/backend/llvm/expr-containers.ts +++ b/packages/compiler/src/backend/llvm/expr-containers.ts @@ -1,3 +1,4 @@ +import { dynDesc } from "../../ir/analysis.js"; /* Focused LLVM expression emission extracted from emitter.ts. */ import { arraySearchHelperName, arraySearchKind } from "./array-search.js"; import { InternalCompilerError } from "../../errors.js"; @@ -49,11 +50,13 @@ import { emitFusedMapSet, rememberMapEntry } from "./map-entry-fusion.js"; import { emitScratchStringSlice, emitStringSliceRead } from "./string-slices.js"; import { STRING_CHAR_AT_DECL, + STRING_CP_AT_BYTE_DECL, STRING_CHAR_AT_INT_MARKER, STRING_CHAR_CODE_AT_DECL, STRING_CHAR_CODE_AT_INT_MARKER, emitAsciiChar, emitAsciiCharCode, + emitStringByteLength, emitUnitCharCode, STRING_UTF16_LEN_DECL, } from "./string-reads.js"; @@ -564,16 +567,30 @@ export function emitStrIntrinsic( return call("scr_str_is_well_formed", "zeroext i1 (ptr)", `ptr ${r.name}`, "i1", false); case "toWellFormed": return call("scr_str_to_well_formed", "ptr (ptr)", `ptr ${r.name}`, "ptr", true); - case "cpAt": - // The code point AT an index as a one-code-point string (+1) — - // the string-for-of desugar's read. - return call( - "scr_str_cp_at", - "ptr (ptr, double)", - `ptr ${r.name}, double ${args[0]!.name}`, - "ptr", - true, - ); + case "byteLength": { + const t = B.tmp(); + // A byte count fits in a signed size, whose conversion is one instruction. + B.line(`${t} = sitofp ${host.sizeType} ${emitStringByteLength(host, r.name)} to double`); + return { name: t, type: e.type }; + } + case "cpAtByte": { + // sc_str_cp_at_byte (string-reads.ts) answers ASCII bytes from the + // charAt table inline. The cursor is a character start, an exact + // integer below the byte count. + host.declare(STRING_CHAR_AT_DECL); + host.declare(STRING_CP_AT_BYTE_DECL); + const index = e.args[0] ? exactInteger(host, args[0]!, e.args[0]) : null; + let wide: string; + if (index) { + wide = widenInteger(host, index); + } else { + wide = B.tmp(); + B.line(`${wide} = fptosi double ${args[0]!.name} to i64`); + } + const t = B.tmp(); + B.line(`${t} = call ptr @sc_str_cp_at_byte(ptr ${r.name}, i64 ${wide})`); + return host.own({ name: t, type: e.type }); + } default: { const _exhaustive: never = method; void _exhaustive; @@ -790,6 +807,9 @@ export function emitArrIntrinsic( case "shift": { host.emitPublishedArrayGuard(r.name, e.receiver.type, method === "pop" ? 2 : 6); const checkedValue = elem.kind === "dyn" && e.type.kind === "dyn"; + // A foreign element (SCR_ARR_FOREIGN) cannot be the typed result: + // removing one throws the checked conversion's TypeError. + const foreignElements = !checkedValue && host.mayHoldForeign(elem); if (!checkedValue && e.type.kind !== "union") throw new InternalCompilerError("llvm emitter bug: array removal result is not a union"); const def = e.type.kind === "union" ? host.unionsById.get(e.type.unionId) : undefined; @@ -832,7 +852,25 @@ export function emitArrIntrinsic( ); B.line(`${slowRaw} = load i64, ptr ${rawSlot}`); B.line(`${has} = icmp eq i8 ${state}, 1`); - B.condBr(has, lp, la); + const lf = foreignElements ? B.newLabel("remove.f") : la; + B.condBr(has, lp, lf); + if (foreignElements) { + // Only the runtime call removes a foreign element; its value moves + // out with the slot, and the throw releases it. + B.startBlock(lf); + const foreign = B.tmp(); + const throwLabel = B.newLabel("remove.foreign"); + B.line(`${foreign} = icmp eq i8 ${state}, 3`); + B.condBr(foreign, throwLabel, la); + B.startBlock(throwLabel); + const payload = B.tmp(); + B.line(`${payload} = inttoptr i64 ${slowRaw} to ptr`); + host.declare(`declare void @scr_dyn_foreign_throw(ptr, ptr)`); + const want = host.cstr(dynDesc(e.type, host.recordsById, host.unionsById)); + B.line(`call void @scr_dyn_foreign_throw(ptr ${payload}, ptr ${want})`); + host.emitAlwaysPendingUnwind(); + B.br(la); + } B.startBlock(lp); let raw = slowRaw; const value = B.tmp(); diff --git a/packages/compiler/src/backend/llvm/expr-primitives.ts b/packages/compiler/src/backend/llvm/expr-primitives.ts index d3b72897e..eed138fe2 100644 --- a/packages/compiler/src/backend/llvm/expr-primitives.ts +++ b/packages/compiler/src/backend/llvm/expr-primitives.ts @@ -661,6 +661,29 @@ export function regexLiteralTemplate( return `@${re.sym}`; } +/** The build-time replacer of a global substitution of regex literal `e` + * with the literal `replacement` (regex-matchers.ts), registered on first + * use. */ +export function regexLiteralReplacer( + host: LlvmEmitterContext, + e: IrExpr & { kind: "regexLit" }, + replacement: string, +): string { + const key = `${e.flags}/${e.pattern}\0${replacement}`; + let rr = host.regexReplacers.get(key); + if (!rr) { + rr = { + sym: `sc_rr_${host.regexReplacers.size}`, + pattern: e.pattern, + flags: e.flags, + replacement, + }; + host.regexReplacers.set(key, rr); + } + host.journal?.entry("regexReplacer", key, rr); + return `@${rr.sym}`; +} + /** A string equality or comparison between `s.charAt(i)` and a literal of * exactly one UTF-16 code unit: the charAt call, the unit, and whether the * charAt is the left operand. */ diff --git a/packages/compiler/src/backend/llvm/expr-records.ts b/packages/compiler/src/backend/llvm/expr-records.ts index 59ab625e5..51688eafa 100644 --- a/packages/compiler/src/backend/llvm/expr-records.ts +++ b/packages/compiler/src/backend/llvm/expr-records.ts @@ -6,7 +6,8 @@ import { LlvmUnsupportedError } from "./unsupported.js"; import type { LlvmEmitterContext, LlValue } from "./expr-context.js"; import { f64Lit, llvmCommentText } from "./common.js"; import { borrowableInputs, emitBorrowedInputs, emitBorrowedInput } from "./borrowed-inputs.js"; -import { regexLiteralTemplate } from "./expr-primitives.js"; +import { regexLiteralReplacer, regexLiteralTemplate } from "./expr-primitives.js"; +import { canReplaceWith, regexLiteralCompiles } from "./regex-matchers.js"; import { isScalarUnionDef, loadInlineUnion, presentBits } from "./inline-unions.js"; import { preservesRegexInputs } from "./reference-effects.js"; @@ -160,6 +161,39 @@ function directRegexOperandMethod(method: string): boolean { } } +/** The build-time replacer of a substitution: a regex literal running on + * its template, with a literal replacement the replacer can write as is. + * Null when the substitution takes the runtime's loop. */ +function literalReplacer( + host: LlvmEmitterContext, + literal: IrExpr | null, + template: string | null, + replacement: IrExpr, +): string | null { + if (literal === null || template === null || literal.kind !== "regexLit") return null; + if (replacement.kind !== "strLit") return null; + if (!canReplaceWith(literal.pattern, literal.flags, replacement.value)) return null; + if (!regexLiteralCompiles(literal.pattern, literal.flags)) return null; + return regexLiteralReplacer(host, literal, replacement.value); +} + +/** scr_regex_replace_with: the runtime runs `replacer` on an ASCII subject + * and its own loop on any other. */ +function emitReplaceWith( + host: LlvmEmitterContext, + e: IrExpr, + r: LlValue, + args: LlValue[], + replacer: string, +): LlValue { + host.declare(`declare ptr @scr_regex_replace_with(ptr, ptr, ptr, ptr)`); + const t = host.B.tmp(); + host.B.line( + `${t} = call ptr @scr_regex_replace_with(ptr ${r.name}, ptr ${args[0]!.name}, ptr ${args[1]!.name}, ptr ${replacer})`, + ); + return host.own({ name: t, type: e.type }); +} + export function emitRegexIntrinsic( host: LlvmEmitterContext, e: IrExpr & { kind: "regexIntrinsic" }, @@ -234,11 +268,16 @@ export function emitRegexIntrinsic( "scr_regex_match_all_captures_into", `ptr ${r.name}, ptr ${args[0]!.name}, ptr ${args[1]!.name}`, ); - case "replaceAll": + case "replaceAll": { + // A global literal (the only kind a replacer is built for) never takes + // replaceAll's non-global TypeError. + const replacer = literalReplacer(host, literal, template, e.args[1]!); + if (replacer !== null) return emitReplaceWith(host, e, r, args, replacer); return fallible( "scr_regex_replace_all", `ptr ${r.name}, ptr ${args[0]!.name}, ptr ${args[1]!.name}`, ); + } case "split": host.declare(`declare ptr @scr_regex_split_limit(ptr, ptr, double)`); { @@ -323,6 +362,8 @@ export function emitRegexIntrinsic( return host.own({ name: t, type: e.type }); } case "replace": { + const replacer = literalReplacer(host, literal, template, e.args[1]!); + if (replacer !== null) return emitReplaceWith(host, e, r, args, replacer); host.declare(`declare ptr @scr_regex_replace(ptr, ptr, ptr)`); const t = B.tmp(); B.line( diff --git a/packages/compiler/src/backend/llvm/expr-stream-bridges.ts b/packages/compiler/src/backend/llvm/expr-stream-bridges.ts index 57172068f..f711795d3 100644 --- a/packages/compiler/src/backend/llvm/expr-stream-bridges.ts +++ b/packages/compiler/src/backend/llvm/expr-stream-bridges.ts @@ -184,7 +184,11 @@ export function streamTypedRefCommitAdapter( } if (t.kind === "array") { const commit = `${snapshot}_commit`; - const check = host.dyn.dynCheckHelper(t); + // A value the element type cannot hold stays in the array as a foreign + // element where the program can store one (LlEmitter.mayHoldForeign). + const check = host.mayHoldForeign(t.elem) + ? host.dyn.dynViewArrayHelper(t) + : host.dyn.dynCheckHelper(t); host.declare(`declare void @scr_arr_release(ptr)`); host.declare(`declare void @scr_arr_outline(ptr)`); host.declare(`declare void @scr_arr_adopt_view_properties(ptr, ptr, ptr)`); @@ -1235,6 +1239,21 @@ export function streamTypedRefMaterializeAdapter( B.line(`call void @scr_dyn_arr_push_hole(ptr ${out})`); B.br(doneLabel); B.startBlock(undefinedLabel); + if (host.mayHoldForeign(elem)) { + // A foreign element (SCR_ARR_FOREIGN) holds its own value. + const foreign = B.tmp(); + const foreignLabel = B.newLabel("live.array.foreign"); + const presentUndefined = B.newLabel("live.array.present.undefined"); + B.line(`${foreign} = fcmp oeq double ${state}, 3.0`); + B.condBr(foreign, foreignLabel, presentUndefined); + B.startBlock(foreignLabel); + host.declare(`declare ptr @scr_arr_get_foreign(ptr, double)`); + const payload = B.tmp(); + B.line(`${payload} = call ptr @scr_arr_get_foreign(ptr %p, double ${index}) ; +1`); + B.line(`call void @scr_dyn_arr_push(ptr ${out}, ptr ${payload})`); + B.br(doneLabel); + B.startBlock(presentUndefined); + } const undefinedValue = B.tmp(); const retained = B.tmp(); B.line(`${undefinedValue} = call ptr @scr_dyn_undefined()`); @@ -1333,7 +1352,16 @@ function liveArrayOps( const refuse = S.newLabel("live.array.refuse"); S.condBr(fits, store, refuse); S.startBlock(refuse); - S.terminate(`ret i1 false`); + if (host.mayHoldForeign(elem)) { + // The array keeps a value its element type cannot hold as a foreign + // element, and undefined as a present undefined. + host.declare(`declare zeroext i1 @scr_arr_set_foreign(ptr, double, ptr)`); + const kept = S.tmp(); + S.line(`${kept} = call zeroext i1 @scr_arr_set_foreign(ptr %p, double %i, ptr %v)`); + S.terminate(`ret i1 ${kept}`); + } else { + S.terminate(`ret i1 false`); + } S.startBlock(store); const next = S.tmp(); S.line( diff --git a/packages/compiler/src/backend/llvm/js-ic.ts b/packages/compiler/src/backend/llvm/js-ic.ts index 11b894a60..d51b838bb 100644 --- a/packages/compiler/src/backend/llvm/js-ic.ts +++ b/packages/compiler/src/backend/llvm/js-ic.ts @@ -406,9 +406,13 @@ function emitGet(host: Host, e: ExprOf<"call">, callee: IrFunction, key: string) inlineLoad(host, receiver.name, ic, INLINE_WAYS, INLINE_PROTO, (cell) => cell, hits, hit, probe); B.startBlock(probe); // The read's probe also answers keys the receiver's plain chain lacks. - host.declare(`declare ptr @scr_js_get_probe(ptr, ptr, ptr)`); + // Only `length` and `next` reads can be answered for a native array or + // iterator, so only those sites take the probe that tests them. + const probeFn = + key === "length" || key === "next" ? "scr_js_get_probe_native" : "scr_js_get_probe"; + host.declare(`declare ptr @${probeFn}(ptr, ptr, ptr)`); const probed = B.tmp(); - B.line(`${probed} = call ptr @scr_js_get_probe(ptr ${receiver.name}, ptr ${name}, ptr ${ic})`); + B.line(`${probed} = call ptr @${probeFn}(ptr ${receiver.name}, ptr ${name}, ptr ${ic})`); const found = B.tmp(); B.line(`${found} = icmp ne ptr ${probed}, null`); const probeEnd = B.currentLabel(); diff --git a/packages/compiler/src/backend/llvm/lib-dispatch.ts b/packages/compiler/src/backend/llvm/lib-dispatch.ts index 95b8bfa2c..b80c5151d 100644 --- a/packages/compiler/src/backend/llvm/lib-dispatch.ts +++ b/packages/compiler/src/backend/llvm/lib-dispatch.ts @@ -1143,6 +1143,8 @@ export function emitLibCall(host: LlvmEmitterContext, e: LibCallExpr): LlValue { case "blob": case "wasm": case "array": + if (e.fn === "array.foreignThrow") return emitArrayForeignThrow(host, e); + return host.emitGenericLibCall(e); case "collection": case "record": case "disposable": @@ -1166,6 +1168,29 @@ export function emitLibCall(host: LlvmEmitterContext, e: LibCallExpr): LlValue { } } +/** `array.foreignThrow`: the checked conversion's TypeError for a foreign + * element read into a typed result. Always throws; the result is a dummy + * of the replaced read's type (the error.argTypeThrow pattern). */ +function emitArrayForeignThrow(host: LlvmEmitterContext, e: LibCallExpr): LlValue { + const B = host.B; + const arr = host.emitExpr(e.args[0]!); + const index = host.emitExpr(e.args[1]!); + const want = host.emitExpr(e.args[2]!); + host.declare(`declare void @scr_arr_foreign_throw(ptr, double, ptr)`); + B.line( + `call void @scr_arr_foreign_throw(ptr ${arr.name}, double ${index.name}, ptr ${want.name})`, + ); + const ty = host.llType(e.type); + if (ty === "void") { + host.emitAlwaysPendingUnwind(); + return { name: "", type: e.type }; + } + const dummy = ty === "double" ? "0.0" : ty === "i1" ? "false" : "null"; + const out = host.own({ name: dummy, type: e.type }); + host.emitAlwaysPendingUnwind(); + return out; +} + /** The settlement adapter of Promise.all's tuple form for the builder * `fnName`, which fulfills the result with the builder's tuple. */ function allTupleAdapterFor(host: LlvmEmitterContext, fnName: string, tuple: IrType): string { diff --git a/packages/compiler/src/backend/llvm/lib-network.ts b/packages/compiler/src/backend/llvm/lib-network.ts index d49bbde14..35eb885cd 100644 --- a/packages/compiler/src/backend/llvm/lib-network.ts +++ b/packages/compiler/src/backend/llvm/lib-network.ts @@ -309,6 +309,7 @@ export function emitNetworkHttpLibCall(host: LlvmEmitterContext, e: LibCallExpr) e.fn === "net.sockOnClose" || e.fn === "net.sockOnConnect" || e.fn === "net.sockOnTimeout" || + e.fn === "net.sockOnDrain" || e.fn === "net.sockOnReadable" || e.fn === "net.sockOnReady" || e.fn === "http.reqOnEnd" || @@ -329,6 +330,7 @@ export function emitNetworkHttpLibCall(host: LlvmEmitterContext, e: LibCallExpr) "net.sockOnClose": "scr_net_sock_on_close", "net.sockOnConnect": "scr_net_sock_on_connect", "net.sockOnTimeout": "scr_net_sock_on_timeout", + "net.sockOnDrain": "scr_net_sock_on_drain", "net.sockOnReadable": "scr_net_sock_on_readable", "net.sockOnReady": "scr_net_sock_on_ready", "http.reqOnEnd": "scr_http_req_on_end", diff --git a/packages/compiler/src/backend/llvm/lib-shared.ts b/packages/compiler/src/backend/llvm/lib-shared.ts index add16707f..49f88c125 100644 --- a/packages/compiler/src/backend/llvm/lib-shared.ts +++ b/packages/compiler/src/backend/llvm/lib-shared.ts @@ -104,6 +104,7 @@ export const LIB_FN_SYMS: Record = { "str.localeCase": "scr_str_locale_case", "str.replaceLiteral": "scr_str_replace_literal", "str.replaceChars": "scr_str_replace_chars", + "str.codePoints": "scr_str_code_points", "num.fromString": "scr_string_to_number", "math.round": "scr_math_round", "math.sin": "sin", @@ -1339,6 +1340,8 @@ export const LIB_FN_SYMS: Record = { "net.sockResetAndDestroy": "scr_net_sock_reset_and_destroy", "net.sockAddress": "scr_net_sock_address", "net.sockBytesWritten": "scr_net_sock_bytes_written", + "net.sockWritableLength": "scr_net_sock_writable_length", + "net.sockWritableNeedDrain": "scr_net_sock_writable_need_drain", "net.sockAddrField": "scr_net_sock_addr_field", "net.sockBytesRead": "scr_net_sock_bytes_read", "net.sockConnecting": "scr_net_sock_connecting", @@ -1663,6 +1666,7 @@ export const LIB_FN_SYMS: Record = { "regex.groupParseInt": "scr_regex_group_parse_int", "regex.resultProp": "scr_regex_result_prop", "array.namedKeys": "scr_arr_named_keys", + "array.foreign": "scr_arr_get_foreign", "array.hasNamedKey": "scr_arr_has_named_key", "array.hasProperties": "scr_arr_has_properties", "array.undefinedKey": "scr_arr_undefined_key", diff --git a/packages/compiler/src/backend/llvm/reference-writes.ts b/packages/compiler/src/backend/llvm/reference-writes.ts index 89132a715..c37af81d4 100644 --- a/packages/compiler/src/backend/llvm/reference-writes.ts +++ b/packages/compiler/src/backend/llvm/reference-writes.ts @@ -107,6 +107,7 @@ const INERT_LIB_CALLS = new Set([ "regex.groupParseInt", "regex.resultProp", "array.namedKeys", + "array.foreign", "array.hasNamedKey", "array.hasProperties", "process.cwd", diff --git a/packages/compiler/src/backend/llvm/regex-matchers.test.ts b/packages/compiler/src/backend/llvm/regex-matchers.test.ts index d009de7c9..5f93f7295 100644 --- a/packages/compiler/src/backend/llvm/regex-matchers.test.ts +++ b/packages/compiler/src/backend/llvm/regex-matchers.test.ts @@ -1,5 +1,6 @@ import { describe, expect, test } from "vitest"; import { + canReplaceWith, compileRegexMatcher, regexLiteralBailReason, regexLiteralCaptures, @@ -152,3 +153,46 @@ describe("regex literal matcher code", () => { expect(text).not.toContain("constant [128 x i8]"); }); }); + +describe("regex literal replacers", () => { + test("take global, non-sticky, non-plain-text patterns and ASCII replacements without $", () => { + expect(canReplaceWith("a[bc]", "g", "<2>")).toBe(true); + expect(canReplaceWith("a[bc]", "gi", "")).toBe(true); + expect(canReplaceWith("a[bc]", "", "x")).toBe(false); + expect(canReplaceWith("a[bc]", "gy", "x")).toBe(false); + expect(canReplaceWith("a[bc]", "g", "$1")).toBe(false); + expect(canReplaceWith("a[bc]", "g", "é")).toBe(false); + // Plain text is a string search at run time. + expect(canReplaceWith("&", "g", "&")).toBe(false); + expect(canReplaceWith("a\\.b", "g", "x")).toBe(false); + // Letters under i are not plain text. + expect(canReplaceWith("ab", "gi", "x")).toBe(true); + }); + + test("append the gap and the replacement and continue the search", () => { + const code = compileRegexMatcher("r", "<[^>]*>", "g", "i64", true, "|"); + const text = code!.lines.join("\n"); + expect(text).toContain('@r.rep = private unnamed_addr constant [1 x i8] c"|"'); + expect(text).toContain("call void @scr_re_replace_out_reserve(ptr %out, i64 %m.need)"); + expect(text).toContain("load <16 x i8>, ptr %m.src, align 1"); + expect(text).toMatch(/br i1 %m\.past, label %nomatch, label %scan/); + expect(text).not.toContain("m.ret:"); + expect(text).toContain("ret i32 %n.found"); + // The search after a match is a 16-byte scan, not a memchr call. + expect(text).not.toContain("@memchr"); + expect(code!.decls).toContain("declare void @scr_re_replace_out_reserve(ptr, i64)"); + // Without vectors the gap is a memcpy. + const scalar = compileRegexMatcher("r", "<[^>]*>", "g", "i64", false, "|")!.lines.join("\n"); + expect(scalar).not.toContain("<16 x i8>"); + expect(scalar).toContain( + "call void @llvm.memcpy.p0.p0.i64(ptr %m.dst, ptr %m.src, i64 %m.gapw, i1 false)", + ); + }); + + test("escape the replacement bytes and skip the copy of an empty one", () => { + const quoted = compileRegexMatcher("r", "a[bc]", "g", "i64", true, '"\\\n')!.lines.join("\n"); + expect(quoted).toContain('c"\\22\\5C\\0A"'); + const empty = compileRegexMatcher("r", "a[bc]", "g", "i64", true, "")!.lines.join("\n"); + expect(empty).not.toContain("@r.rep"); + }); +}); diff --git a/packages/compiler/src/backend/llvm/regex-matchers.ts b/packages/compiler/src/backend/llvm/regex-matchers.ts index d83d6bae5..0491a6446 100644 --- a/packages/compiler/src/backend/llvm/regex-matchers.ts +++ b/packages/compiler/src/backend/llvm/regex-matchers.ts @@ -24,6 +24,8 @@ * `len`, it returns the capture count (whole match included) so the runtime * can check it against the compiled bytecode. */ +import { InternalCompilerError } from "../../errors.js"; + const ASCII = 128; const INF = -1; /** Larger patterns keep the interpreter (binary size). */ @@ -1853,10 +1855,13 @@ export function compileRegexMatcher( flags: string, sizeType: string, vector: boolean, + replacement: string | null = null, ): RegexMatcherCode | null { const parsed = parseSubset(pattern, flags); if (parsed === null) return null; const sticky = flags.includes("y"); + if (replacement !== null && !canReplaceWith(pattern, flags, replacement)) + throw new InternalCompilerError(`llvm emitter bug: no replacer for /${pattern}/${flags}`); const { root, ncap } = parsed; const gen = new RxGen(sym, ncap, vector); const end: FirstInfo = { set: emptySet(), nullable: true, bol: false, uncond: true }; @@ -1873,6 +1878,12 @@ export function compileRegexMatcher( fn.push(` %stk = alloca ptr`); fn.push(` %sbuf = alloca [64 x i32]`); for (let r = 0; r < gen.regs; r++) fn.push(` %r${r} = alloca i32`); + if (replacement !== null) { + fn.push(` %next = alloca i32`); + fn.push(` %nfound = alloca i32`); + fn.push(` store i32 0, ptr %next`); + fn.push(` store i32 0, ptr %nfound`); + } fn.push(` store ptr %sbuf, ptr %stk`); fn.push(` store i32 64, ptr %scap`); fn.push(` store i32 %start, ptr %at`); @@ -1900,7 +1911,9 @@ export function compileRegexMatcher( } else { fn.push(` br label %seek`); } - if (firstCount === 1) { + // A replacer seeks again after every match, often a few bytes on: the + // 16-byte scan avoids a memchr call per match. + if (firstCount === 1 && replacement === null) { fn.push(`seek:`); const byte = intervals(first.set)[0]![0]!; fn.push(` %rest = sub i32 %len, %a`); @@ -1944,6 +1957,98 @@ export function compileRegexMatcher( fn.push(` br i1 %over, label %nomatch, label %scan`); } + if (replacement !== null) { + emitReplaceMatch(fn, sym, replacement, vector); + } else { + emitReturnMatch(fn, ncap); + } + + const result = replacement !== null ? "%n.found" : "0"; + fn.push(`nomatch:`); + if (replacement !== null) { + fn.push(` %n.next = load i32, ptr %next`); + fn.push(` %n.nextp = getelementptr inbounds i8, ptr %out, i64 16`); + fn.push(` %n.nextw = sext i32 %n.next to i64`); + fn.push(` store i64 %n.nextw, ptr %n.nextp`); + fn.push(` %n.found = load i32, ptr %nfound`); + } + fn.push(` %n.stk = load ptr, ptr %stk`); + fn.push(` %n.heap = icmp ne ptr %n.stk, %sbuf`); + fn.push(` br i1 %n.heap, label %n.free, label %n.ret`); + fn.push(`n.free:`); + fn.push(` call void @scr_regex_stack_free(ptr %n.stk)`); + fn.push(` br label %n.ret`); + fn.push(`n.ret:`); + fn.push(` ret i32 ${result}`); + + gen.emitTail(); + for (const l of gen.body) fn.push(l); + fn.push(`}`); + + const decls = REGEX_MATCHER_DECLS.slice(); + if (gen.body.some((l) => l.includes("@llvm.cttz.i16"))) + decls.push(`declare i16 @llvm.cttz.i16(i16, i1 immarg)`); + if (!sticky && !first.nullable && firstCount === 1 && replacement === null) + decls.push(`declare ptr @memchr(ptr, i32, ${sizeType})`); + if (replacement !== null) { + decls.push(`declare void @scr_re_replace_out_reserve(ptr, i64)`); + decls.push(`declare void @llvm.memcpy.p0.p0.i64(ptr, ptr, i64, i1 immarg)`); + } + const globals = [...gen.globals]; + if (replacement !== null && replacement.length > 0) + globals.push( + `@${sym}.rep = private unnamed_addr constant [${replacement.length} x i8] c"${llvmBytes(replacement)}"`, + ); + return { lines: [...globals, ...fn], decls }; +} + +/** Whether a substitution can run through a build-time replacer: a global, + * non-sticky pattern that is not plain text (the runtime searches those as + * strings) and an ASCII replacement without `$` patterns. */ +export function canReplaceWith(pattern: string, flags: string, replacement: string): boolean { + if (!flags.includes("g") || flags.includes("y") || isPlainTextPattern(pattern, flags)) + return false; + for (let i = 0; i < replacement.length; i++) { + const c = replacement.charCodeAt(i); + if (c >= ASCII || c === 0x24) return false; + } + return true; +} + +/** scr_regex_plain_text's rule: a pattern of literal characters and escaped + * metacharacters (no letters under `i`) that the runtime searches as text. */ +function isPlainTextPattern(pattern: string, flags: string): boolean { + const fold = flags.includes("i"); + if (pattern.length === 0 || pattern.length > 256) return false; + for (let i = 0; i < pattern.length; i++) { + let c = pattern[i]!; + if (c === "\\") { + if (i + 1 >= pattern.length) return false; + c = pattern[++i]!; + if (!"^$\\.*+?()[]{}|/".includes(c)) return false; + } else { + const code = c.charCodeAt(0); + if (code < 0x20 || code >= 0x7f || "^$.*+?()[]{}|".includes(c)) return false; + } + if (fold && /[a-z]/i.test(c)) return false; + } + return true; +} + +function llvmBytes(text: string): string { + let out = ""; + for (let i = 0; i < text.length; i++) { + const c = text.charCodeAt(i); + out += + c >= 0x20 && c < 0x7f && c !== 0x22 && c !== 0x5c + ? text[i] + : `\\${c.toString(16).toUpperCase().padStart(2, "0")}`; + } + return out; +} + +/** The match block of a matcher: store the capture pointers, return 1. */ +function emitReturnMatch(fn: string[], ncap: number): void { fn.push(`match:`); fn.push(` %m.end = load i32, ptr %pos`); fn.push(` %m.start = load i32, ptr %at`); @@ -1969,25 +2074,72 @@ export function compileRegexMatcher( fn.push(` br label %m.ret`); fn.push(`m.ret:`); fn.push(` ret i32 1`); +} - fn.push(`nomatch:`); - fn.push(` %n.stk = load ptr, ptr %stk`); - fn.push(` %n.heap = icmp ne ptr %n.stk, %sbuf`); - fn.push(` br i1 %n.heap, label %n.free, label %n.ret`); - fn.push(`n.free:`); - fn.push(` call void @scr_regex_stack_free(ptr %n.stk)`); - fn.push(` br label %n.ret`); - fn.push(`n.ret:`); - fn.push(` ret i32 0`); - - gen.emitTail(); - for (const l of gen.body) fn.push(l); - fn.push(`}`); - - const decls = REGEX_MATCHER_DECLS.slice(); - if (gen.body.some((l) => l.includes("@llvm.cttz.i16"))) - decls.push(`declare i16 @llvm.cttz.i16(i16, i1 immarg)`); - if (!sticky && !first.nullable && firstCount === 1) - decls.push(`declare ptr @memchr(ptr, i32, ${sizeType})`); - return { lines: [...gen.globals, ...fn], decls }; +/** The match block of a replacer: append the gap since the previous match + * and the replacement to the output (ScrReReplaceOut: len, cap, next at + * 0, 8 and 16, then data), and continue the search at the match end, or one past it for an + * empty match, as the runtime's replace loop does. */ +function emitReplaceMatch(fn: string[], sym: string, replacement: string, vector: boolean): void { + const n = replacement.length; + fn.push(`match:`); + fn.push(` %m.end = load i32, ptr %pos`); + fn.push(` %m.start = load i32, ptr %at`); + fn.push(` %m.next = load i32, ptr %next`); + fn.push(` %m.gap = sub i32 %m.start, %m.next`); + fn.push(` %m.gapw = zext i32 %m.gap to i64`); + fn.push(` %m.lenp = getelementptr inbounds i8, ptr %out, i64 0`); + fn.push(` %m.capp = getelementptr inbounds i8, ptr %out, i64 8`); + fn.push(` %m.datap = getelementptr inbounds i8, ptr %out, i64 24`); + fn.push(` %m.len0 = load i64, ptr %m.lenp`); + fn.push(` %m.cap = load i64, ptr %m.capp`); + fn.push(` %m.room = sub i64 %m.cap, %m.len0`); + // 16 bytes of slack past the gap and the replacement for the 16-byte copy. + fn.push(` %m.need = add i64 %m.gapw, ${n + 32}`); + fn.push(` %m.fits = icmp uge i64 %m.room, %m.need`); + fn.push(` br i1 %m.fits, label %m.put, label %m.grow`); + fn.push(`m.grow:`); + fn.push(` call void @scr_re_replace_out_reserve(ptr %out, i64 %m.need)`); + fn.push(` br label %m.put`); + fn.push(`m.put:`); + fn.push(` %m.data = load ptr, ptr %m.datap`); + fn.push(` %m.len = load i64, ptr %m.lenp`); + fn.push(` %m.dst = getelementptr inbounds i8, ptr %m.data, i64 %m.len`); + fn.push(` %m.nextw = sext i32 %m.next to i64`); + fn.push(` %m.src = getelementptr inbounds i8, ptr %s, i64 %m.nextw`); + if (vector) { + // A gap of at most 16 bytes with 16 readable subject bytes is one + // unaligned 16-byte copy. + fn.push(` %m.small = icmp ule i32 %m.gap, 16`); + fn.push(` %m.rest = sub i32 %len, %m.next`); + fn.push(` %m.readable = icmp sge i32 %m.rest, 16`); + fn.push(` %m.fast = and i1 %m.small, %m.readable`); + fn.push(` br i1 %m.fast, label %m.copy16, label %m.copy`); + fn.push(`m.copy16:`); + fn.push(` %m.v = load <16 x i8>, ptr %m.src, align 1`); + fn.push(` store <16 x i8> %m.v, ptr %m.dst, align 1`); + fn.push(` br label %m.rep`); + } else { + fn.push(` br label %m.copy`); + } + fn.push(`m.copy:`); + fn.push(` call void @llvm.memcpy.p0.p0.i64(ptr %m.dst, ptr %m.src, i64 %m.gapw, i1 false)`); + fn.push(` br label %m.rep`); + fn.push(`m.rep:`); + fn.push(` %m.rdst = getelementptr inbounds i8, ptr %m.dst, i64 %m.gapw`); + if (n > 0) + fn.push(` call void @llvm.memcpy.p0.p0.i64(ptr %m.rdst, ptr @${sym}.rep, i64 ${n}, i1 false)`); + fn.push(` %m.len1 = add i64 %m.len, %m.gapw`); + fn.push(` %m.len2 = add i64 %m.len1, ${n}`); + fn.push(` store i64 %m.len2, ptr %m.lenp`); + fn.push(` store i32 %m.end, ptr %next`); + fn.push(` %m.found0 = load i32, ptr %nfound`); + fn.push(` %m.found1 = add i32 %m.found0, 1`); + fn.push(` store i32 %m.found1, ptr %nfound`); + fn.push(` %m.empty = icmp eq i32 %m.end, %m.start`); + fn.push(` %m.after = add i32 %m.end, 1`); + fn.push(` %m.at = select i1 %m.empty, i32 %m.after, i32 %m.end`); + fn.push(` store i32 %m.at, ptr %at`); + fn.push(` %m.past = icmp sgt i32 %m.at, %len`); + fn.push(` br i1 %m.past, label %nomatch, label %scan`); } diff --git a/packages/compiler/src/backend/llvm/shapes.test.ts b/packages/compiler/src/backend/llvm/shapes.test.ts index 61c1a074d..d9e5fa641 100644 --- a/packages/compiler/src/backend/llvm/shapes.test.ts +++ b/packages/compiler/src/backend/llvm/shapes.test.ts @@ -146,11 +146,19 @@ describe("LLVM runtime RC symbols", () => { // The dyn fast path buffers surviving values by kind, as scr_dyn_release does. const ir = emitInlineRcHelpers(host).join("\n"); expect(ir).toContain("call void @scr_dyn_release_survivor(ptr %o)"); - expect(ir).toMatch( - /switch i32 %kind, label %done \[ i32 4, label %container i32 5, label %container i32 8, label %root/, + // Every kind that can be a candidate takes one path; scalars stay inline. + expect(ir).toContain( + "switch i32 %kind, label %done [ i32 4, label %candidate i32 5, label %candidate i32 8, label %candidate i32 9, label %candidate i32 12, label %candidate i32 13, label %candidate ]", ); - // Arrays and objects not yet buffered ask the runtime, which skips intact documents. + // Values not yet buffered ask the runtime, which decides per value. expect(ir).toMatch(/br i1 %cqueued, label %done, label %survivor/); + // The runtime buffers dyn candidates, so the dyn helper has no inline append. + const dynHelper = ir.slice( + ir.indexOf("@sc_rc_release_dyn("), + ir.indexOf("\n}", ir.indexOf("@sc_rc_release_dyn(")), + ); + expect(dynHelper).not.toContain("sc_cyc_append"); + expect(dynHelper).not.toContain("scr_cyc_on_release"); }); test("sanitized builds call the fast-path helpers out of line", () => { @@ -208,36 +216,42 @@ describe("LLVM runtime RC symbols", () => { }); test("optimized builds buffer candidates through the module's append helper", () => { - const emit = (inlineRc: true | "release", extra: Partial) => { + const emit = (inlineRc: true | "release", extra: Partial, kind: "union" | "dyn") => { const { host } = declarationHost(64, inlineRc); const configured: ShapeHost = { ...host, inlineAlloc: true, ...extra }; - releaseSym(configured, { kind: "dyn" }); + releaseSym(configured, kind === "union" ? ({ kind, unionId: "u0" } as never) : { kind }); return { ir: emitInlineRcHelpers(configured).join("\n"), decls: inlineRcDecls(configured), }; }; + const { ir, decls } = emit(true, {}, "union"); + expect(ir).toContain("call preserve_mostcc void @sc_cyc_append(ptr %o)"); + expect(ir).toContain("define internal preserve_mostcc void @sc_cyc_append(ptr %o) noinline"); + // The young test reads the generation byte at obj-5; the append fills + // scr_cyc_buffers' nursery buffer and the header's position at obj-4. + expect(ir).toContain("getelementptr i8, ptr %o, i64 -5"); + expect(ir).toContain("getelementptr inbounds i8, ptr @scr_cyc_buffers, i64 96"); + expect(ir).toContain("getelementptr inbounds i8, ptr @scr_cyc_buffers, i64 104"); + expect(ir).toContain("load i64, ptr @scr_cyc_live"); + expect(ir).toContain("or i32 %i, -2147483648"); + expect(ir).toContain("call void @scr_cyc_on_release(ptr %o)"); + expect(decls).toContain( + "@scr_cyc_buffers = external global { [3 x { ptr, i64, i64, i64 }], i64, i64 }", + ); + expect(decls).toContain("@scr_cyc_live = external global i64"); + // The dyn release buffers through scr_dyn_release_survivor, so a module + // whose only headered release is dyn's has no append helper. for (const mode of [true, "release"] as const) { - const { ir, decls } = emit(mode, {}); - expect(ir).toContain("call preserve_mostcc void @sc_cyc_append(ptr %o)"); - expect(ir).toContain("define internal preserve_mostcc void @sc_cyc_append(ptr %o) noinline"); - // The young test reads the generation byte at obj-5; the append fills - // scr_cyc_buffers' nursery buffer and the header's position at obj-4. - expect(ir).toContain("getelementptr i8, ptr %o, i64 -5"); - expect(ir).toContain("getelementptr inbounds i8, ptr @scr_cyc_buffers, i64 96"); - expect(ir).toContain("getelementptr inbounds i8, ptr @scr_cyc_buffers, i64 104"); - expect(ir).toContain("load i64, ptr @scr_cyc_live"); - expect(ir).toContain("or i32 %i, -2147483648"); - expect(ir).toContain("call void @scr_cyc_on_release(ptr %o)"); - expect(decls).toContain( - "@scr_cyc_buffers = external global { [3 x { ptr, i64, i64, i64 }], i64, i64 }", - ); - expect(decls).toContain("@scr_cyc_live = external global i64"); + const dyn = emit(mode, {}, "dyn"); + expect(dyn.ir).not.toContain("sc_cyc_append"); + expect(dyn.ir).not.toContain("scr_cyc_on_release"); + expect(dyn.decls.join("\n")).not.toContain("scr_cyc_buffers"); } // Worker executables and targets without inline allocation call // scr_cyc_on_release, which tries the same append first. for (const extra of [{ threadLocalAlloc: true }, { inlineAlloc: false }]) { - const { ir, decls } = emit(true, extra); + const { ir, decls } = emit(true, extra, "union"); expect(ir).not.toContain("sc_cyc_append"); expect(ir).toContain("call void @scr_cyc_on_release(ptr %o)"); expect(decls.join("\n")).not.toContain("scr_cyc_buffers"); diff --git a/packages/compiler/src/backend/llvm/shapes.ts b/packages/compiler/src/backend/llvm/shapes.ts index ab623f934..7498235b0 100644 --- a/packages/compiler/src/backend/llvm/shapes.ts +++ b/packages/compiler/src/backend/llvm/shapes.ts @@ -589,9 +589,10 @@ interface InlineRcFamily { * repeats its own checks, so the fast path never duplicates it. * * The dyn release buffers a surviving value by kind (scr_dyn_release): - * the fast path decides the always and never cases and calls - * scr_dyn_release_survivor for kinds whose candidacy is per value, - * including arrays and objects of intact documents (ScrDynRegion). + * the fast path skips scalars and values already in a candidate buffer or + * never candidates, and calls scr_dyn_release_survivor for the rest, which + * decides per value (intact documents, ScrDynRegion; capsules and handles + * without edges). * * The array release inlines the decrement of an array that stays alive * and is not headered (`a->cyc`): those are most array releases in @@ -698,9 +699,10 @@ function requestRcHelper(host: ShapeHost, family: string, op: RcOp): string { for (const decl of rcHelperDecls(family, op)) host.declare(decl); // Its body is written last (emitInlineRcHelpers), after the requests are // collected: a release that can buffer a candidate requests the append - // helper now. The jsval row (JS_RC_ROW) has no collector header; its - // cells release through dyn's helper, which makes its own request. - if (op === "release" && row.cycle === "always") requestCycleAppend(host); + // helper now. The dyn release buffers through scr_dyn_release_survivor + // instead. The jsval row (JS_RC_ROW) has no collector header; its cells + // release through dyn's helper. + if (op === "release" && row.cycle === "always" && family !== "dyn") requestCycleAppend(host); return `@${inlineRcName(family, op)}`; } @@ -820,8 +822,11 @@ function rcHelperDecls(family: string, op: RcOp): string[] { if (op === "retain" || row.release === null) return []; return [ `declare void @${row.release}(ptr)`, - ...(row.cycle === "always" ? [`declare void @scr_cyc_on_release(ptr)`] : []), - ...(family === "dyn" ? [`declare void @scr_dyn_release_survivor(ptr)`] : []), + ...(family === "dyn" + ? [`declare void @scr_dyn_release_survivor(ptr)`] + : row.cycle === "always" + ? [`declare void @scr_cyc_on_release(ptr)`] + : []), ]; } @@ -931,21 +936,21 @@ function releaseHelper(host: ShapeHost, family: string, row: InlineRcFamily): st // scr_dyn_release buffers a surviving value by kind: always for // functions and proxies, per value for arrays and objects (not // while their document is intact, ScrDynRegion), capsules and - // handles (scr_dyn_release_survivor), never for scalars. An - // array or object already in a candidate buffer stays inline. + // handles, never for scalars. Every kind that can be buffered + // takes one path, which keeps each inlined site small: a value + // already in a candidate buffer (or never one) stays inline, and + // scr_dyn_release_survivor decides the rest. ` %kp = getelementptr i8, ptr %o, ${S} ${S === "i64" ? 8 : 4} ; ScrDyn.kind`, ` %kind = load i32, ptr %kp`, - // SCR_DYN_ARR 4, OBJ 5, FUNC 8, PROXY 13; HANDLE 9, TYPED_REF 12. - ` switch i32 %kind, label %done [ i32 4, label %container i32 5, label %container i32 8, label %root i32 13, label %root i32 9, label %survivor i32 12, label %survivor ]`, - `container:`, + // SCR_DYN_ARR 4, OBJ 5, FUNC 8, HANDLE 9, TYPED_REF 12, PROXY 13. + ` switch i32 %kind, label %done [ i32 4, label %candidate i32 5, label %candidate i32 8, label %candidate i32 9, label %candidate i32 12, label %candidate i32 13, label %candidate ]`, + `candidate:`, ` %cbufp = getelementptr i8, ptr %o, ${S} -${host.cycleBufOffset}`, ` %cbuf = load i32, ptr %cbufp`, ` %cqueued = icmp slt i32 %cbuf, 0`, ` br i1 %cqueued, label %done, label %survivor`, `survivor:`, ` call void @scr_dyn_release_survivor(ptr %o)`, - ` br label %done`, - `root:`, ] : family === "box" ? [ @@ -964,7 +969,9 @@ function releaseHelper(host: ShapeHost, family: string, row: InlineRcFamily): st `root:`, ] : []), - ...(cycle === "always" ? cycleRootLines(host, "done") : [` br label %done`]), + ...(cycle === "always" && family !== "dyn" + ? cycleRootLines(host, "done") + : [` br label %done`]), `done:`, ` ret void`, `}`, diff --git a/packages/compiler/src/backend/llvm/string-lifetimes.test.ts b/packages/compiler/src/backend/llvm/string-lifetimes.test.ts index 07ad31153..72fc57752 100644 --- a/packages/compiler/src/backend/llvm/string-lifetimes.test.ts +++ b/packages/compiler/src/backend/llvm/string-lifetimes.test.ts @@ -341,7 +341,7 @@ const methodCases: { { method: "normalize", args: [ref("argument")], result: STRING, target: "normalize" }, { method: "isWellFormed", args: [], result: BOOL, target: "is_well_formed" }, { method: "toWellFormed", args: [], result: STRING, target: "to_well_formed" }, - { method: "cpAt", args: [num(1)], result: STRING, target: "cp_at" }, + { method: "cpAtByte", args: [num(1)], result: STRING, target: "cp_at_byte", inline: true }, ]; test.each(methodCases)( diff --git a/packages/compiler/src/backend/llvm/string-lifetimes.ts b/packages/compiler/src/backend/llvm/string-lifetimes.ts index 6c3079cc1..5a4504813 100644 --- a/packages/compiler/src/backend/llvm/string-lifetimes.ts +++ b/packages/compiler/src/backend/llvm/string-lifetimes.ts @@ -30,7 +30,8 @@ export function borrowsStringInputs(method: IrStrIntrinsicMethod): boolean { case "normalize": case "isWellFormed": case "toWellFormed": - case "cpAt": + case "byteLength": + case "cpAtByte": return true; default: return false; diff --git a/packages/compiler/src/backend/llvm/string-reads.ts b/packages/compiler/src/backend/llvm/string-reads.ts index 5b7a26c6c..1f77de2c2 100644 --- a/packages/compiler/src/backend/llvm/string-reads.ts +++ b/packages/compiler/src/backend/llvm/string-reads.ts @@ -43,6 +43,10 @@ export const STRING_CHAR_CODE_AT_INT_MARKER = "; uses @sc_str_char_code_at_int"; export const STRING_CHAR_AT_DECL = "declare ptr @scr_str_char_at(ptr, double)"; /** Requests the integer-index charAt helper, as the marker above does. */ export const STRING_CHAR_AT_INT_MARKER = "; uses @sc_str_char_at_int"; +/** The code-point character starting at a byte offset (the string + * iterator's step). Requested together with STRING_CHAR_AT_DECL, which + * defines the ASCII table. */ +export const STRING_CP_AT_BYTE_DECL = "declare ptr @scr_str_cp_at_byte(ptr, i64)"; /** The proven-ASCII flag of string `s`, as an i1. */ export function emitStringAsciiFlag(host: LlvmEmitterContext, s: string): string { @@ -351,6 +355,29 @@ export const STRING_READ_HELPERS: readonly StringReadHelper[] = [ ]; }, }, + { + // The string iterator's step at byte `%b`, a character start below the + // byte count: an ASCII byte is the table entry for it; any other lead + // byte takes the runtime, which copies the character's bytes. + decl: STRING_CP_AT_BYTE_DECL, + define: (sz, attrs) => [ + `define internal ptr @sc_str_cp_at_byte(ptr %s, i64 %b) ${attrs} {`, + `entry:`, + ` %data = getelementptr inbounds %ScrStr, ptr %s, i32 1`, + ` %p = getelementptr inbounds i8, ptr %data, i64 %b`, + ` %byte = load i8, ptr %p`, + ` %ascii = icmp sge i8 %byte, 0`, + ` br i1 %ascii, label %read, label %slow`, + `read:`, + ` %code = zext i8 %byte to i64`, + ` %char = getelementptr inbounds [128 x { ${sz}, ${sz}, ${sz}, [2 x i8] }], ptr @sc_ascii_chars, i64 0, i64 %code`, + ` ret ptr %char`, + `slow:`, + ` %r = call ptr @scr_str_cp_at_byte(ptr %s, i64 %b) nounwind willreturn`, + ` ret ptr %r`, + `}`, + ], + }, { // An exact integer index, as for sc_str_char_code_at_int. Requested // together with STRING_CHAR_AT_DECL, which defines the table. diff --git a/packages/compiler/src/backend/llvm/walkers.ts b/packages/compiler/src/backend/llvm/walkers.ts index 58df3e845..7c67e98ae 100644 --- a/packages/compiler/src/backend/llvm/walkers.ts +++ b/packages/compiler/src/backend/llvm/walkers.ts @@ -26,6 +26,7 @@ import { llFieldType, loadStructField, releaseSym, + retainSym, traceAdapter, type ShapeHost, } from "./shapes.js"; @@ -41,6 +42,8 @@ import { undefinedArmTag } from "../../ir/analysis.js"; * scr_error-style label texts). */ export interface WalkerHost extends ShapeHost { readonly unionsById: Map; + /** Arrays of this element type can hold SCR_ARR_FOREIGN elements. */ + mayHoldForeign(elem: IrType): boolean; /** `@`-ref of an interned immortal ScrStr literal. */ internLiteral(text: string): string; /** `@`-ref of an interned NUL-terminated byte-array constant. */ @@ -325,9 +328,7 @@ export class LlWalkers { if (i > 0) this.putc(B, "%b", "44"); // ',' if (edgeable(f.type)) this.jbEdgeIdx(B, String(i)); const v = this.loadField(B, "%v", shapeId, fieldIndex.get(f.name)!, f.type); - B.line( - `call void @${this.jsonWriteHelper(f.type)}(ptr %b, ${this.valTy(f.type)} ${v}) ; [${llvmCommentText(f.name)}]`, - ); + this.writeField(B, v, f.type, `[${llvmCommentText(f.name)}]`); }); this.putc(B, "%b", "93"); // ']' if (cyclic) this.jbLeave(B); @@ -359,9 +360,7 @@ export class LlWalkers { this.puts(B, "%b", `${i > 0 ? "," : ""}${jsonObjectKeyLabel(f.name)}`); if (edgeable(f.type)) this.jbEdgeProp(B, f.name); const v = this.loadField(B, "%v", shapeId, fieldIndex.get(f.name)!, f.type); - B.line( - `call void @${this.jsonWriteHelper(f.type)}(ptr %b, ${this.valTy(f.type)} ${v}) ; ${llvmCommentText(f.name)}`, - ); + this.writeField(B, v, f.type, `${llvmCommentText(f.name)}`); }); } else { const first = B.slot(); @@ -396,9 +395,7 @@ export class LlWalkers { comma(); this.puts(B, "%b", jsonObjectKeyLabel(f.name)); if (edgeable(f.type)) this.jbEdgeProp(B, f.name); - B.line( - `call void @${this.jsonWriteHelper(f.type)}(ptr %b, ${this.valTy(f.type)} ${v}) ; ${llvmCommentText(f.name)}`, - ); + this.writeField(B, v, f.type, `${llvmCommentText(f.name)}`); if (skip !== null) { B.br(skip); B.startBlock(skip); @@ -520,9 +517,7 @@ export class LlWalkers { B.startBlock(lKey); this.puts(B, "%b", jsonObjectKeyLabel(fieldName)); if (cyclic && traceAdapter(this.host, f.type) !== null) this.jbEdgeProp(B, fieldName); - B.line( - `call void @${this.jsonWriteHelper(f.type)}(ptr %b, ${this.valTy(f.type)} ${v}) ; ${llvmCommentText(fieldName)}`, - ); + this.writeField(B, v, f.type, `${llvmCommentText(fieldName)}`); B.terminate("ret i1 true"); }); B.startBlock(lSkip); @@ -539,14 +534,15 @@ export class LlWalkers { /** Whether the walker for `t` can leave an exception pending: cyclic * values, records serialized through a kept checked source, and - * checked-dynamic values (toJSON methods, getters, BigInts). */ + * checked-dynamic values (toJSON methods, getters, BigInts), including + * the foreign elements of arrays that can hold them. */ jsonWriteMayThrow(t: IrType, seen = new Set()): boolean { if (traceAdapter(this.host, t) !== null) return true; switch (t.kind) { case "dyn": return true; case "array": - return this.jsonWriteMayThrow(t.elem, seen); + return this.host.mayHoldForeign(t.elem) || this.jsonWriteMayThrow(t.elem, seen); case "union": return (this.host.unionsById.get(t.unionId)?.arms ?? []).some((arm) => this.jsonWriteMayThrow(arm, seen), @@ -678,6 +674,47 @@ export class LlWalkers { B.line(`call void @scr_arr_release(ptr ${ks})`); } + /** Serializing a value of `t` can call user code that changes the + * arrays or records holding it: it reaches an array that can hold + * foreign elements (their toJSON and getters), or a value whose own + * serialization may (a class instance, a checked-dynamic value), in a + * program that has foreign elements at all. */ + private serializationMayCallUserCode(t: IrType, seen = new Set()): boolean { + const host = this.host; + if (!host.mayHoldForeign({ kind: "f64" }) && !host.mayHoldForeign({ kind: "string" })) + return false; + const key = typeKey(t); + if (seen.has(key)) return false; + seen.add(key); + switch (t.kind) { + case "dyn": + case "object": + return true; + case "array": + return host.mayHoldForeign(t.elem) || this.serializationMayCallUserCode(t.elem, seen); + case "record": + return (host.recordsById.get(t.shapeId)?.fields ?? []).some((field) => + this.serializationMayCallUserCode(field.type, seen), + ); + case "union": + return (host.unionsById.get(t.unionId)?.arms ?? []).some((arm) => + this.serializationMayCallUserCode(arm, seen), + ); + default: + return false; + } + } + + /** Write a field value loaded borrowed from its record. When its + * serialization can call user code that replaces the field, the value + * is held across the write. */ + private writeField(B: BlockBuilder, v: string, t: IrType, comment: string): void { + const held = isRefCounted(t) && this.serializationMayCallUserCode(t); + if (held) B.line(`call ptr ${retainSym(this.host, t)}(ptr ${v})`); + B.line(`call void @${this.jsonWriteHelper(t)}(ptr %b, ${this.valTy(t)} ${v}) ; ${comment}`); + if (held) B.line(`call void ${releaseSym(this.host, t)}(ptr ${v})`); + } + private emitArrayWriter(B: BlockBuilder, elem: IrType, cyclic: boolean): void { const host = this.host; const w = this.jsonWriteHelper(elem); @@ -687,7 +724,12 @@ export class LlWalkers { if (cyclic) this.jbEnter(B, true); this.putc(B, "%b", "91"); // '[' const finish = B.newLabel("jwa.end"); - if (elem.kind !== "bool") this.emitDenseArrayElements(B, elem, w, cyclic, finish); + // The dense walk borrows the storage and each element. When an + // element's serialization can call user code, which can grow or + // shrink the array or replace the element, the general walk below + // reads each index afresh and holds its element instead. + if (elem.kind !== "bool" && !this.serializationMayCallUserCode(elem)) + this.emitDenseArrayElements(B, elem, w, cyclic, finish); const len = B.tmp(); B.line(`${len} = call double @scr_arr_len(ptr %v)`); B.countedLoop(len, (i) => { @@ -710,6 +752,25 @@ export class LlWalkers { B.line(`${present} = fcmp oeq double ${state}, ${f64Lit(1)}`); B.condBr(present, valueBlock, missingBlock); B.startBlock(missingBlock); + if (host.mayHoldForeign(elem)) { + // A foreign element (SCR_ARR_FOREIGN) serializes its own value, + // after its toJSON runs with the index as its key. + const foreign = B.tmp(); + const foreignBlock = B.newLabel("jwa.foreign"); + const nullBlock = B.newLabel("jwa.null"); + B.line(`${foreign} = fcmp oeq double ${state}, ${f64Lit(3)}`); + B.condBr(foreign, foreignBlock, nullBlock); + B.startBlock(foreignBlock); + host.declare(`declare ptr @scr_arr_get_foreign(ptr, double)`); + host.declare(`declare void @scr_dyn_release(ptr)`); + const payload = B.tmp(); + B.line(`${payload} = call ptr @scr_arr_get_foreign(ptr %v, double ${i}) ; +1`); + host.declare(`declare void @scr_jb_put_dyn_element(ptr, ptr, double)`); + B.line(`call void @scr_jb_put_dyn_element(ptr %b, ptr ${payload}, double ${i})`); + B.line(`call void @scr_dyn_release(ptr ${payload})`); + B.br(doneBlock); + B.startBlock(nullBlock); + } host.declare(`declare void @scr_jb_puts(ptr, ptr)`); B.line(`call void @scr_jb_puts(ptr %b, ptr ${host.cstr("null")})`); B.br(doneBlock); @@ -911,7 +972,28 @@ export class LlWalkers { const lvalue = B.newLabel("uj.p"); B.line(`${state} = call double @scr_arr_state(ptr %a, double ${i})`); B.line(`${isValue} = fcmp oeq double ${state}, ${f64Lit(1)}`); - B.condBr(isValue, lvalue, next); + if (host.mayHoldForeign({ kind: "union", unionId })) { + // A foreign element (SCR_ARR_FOREIGN) joins as its ToString, like + // every element but undefined and null. + const notValue = B.newLabel("uj.f"); + const foreignLabel = B.newLabel("uj.foreign"); + B.condBr(isValue, lvalue, notValue); + B.startBlock(notValue); + const foreign = B.tmp(); + B.line(`${foreign} = fcmp oeq double ${state}, ${f64Lit(3)}`); + B.condBr(foreign, foreignLabel, next); + B.startBlock(foreignLabel); + host.declare(`declare ptr @scr_arr_get_foreign(ptr, double)`); + host.declare(`declare void @scr_dyn_release(ptr)`); + const payload = B.tmp(); + B.line(`${payload} = call ptr @scr_arr_get_foreign(ptr %a, double ${i}) ; +1`); + host.declare(`declare void @scr_jb_join_dyn(ptr, ptr)`); + B.line(`call void @scr_jb_join_dyn(ptr ${buf}, ptr ${payload})`); + B.line(`call void @scr_dyn_release(ptr ${payload})`); + B.br(next); + } else { + B.condBr(isValue, lvalue, next); + } B.startBlock(lvalue); const u = B.tmp(); B.line(`${u} = call ptr @scr_arr_peek_ref(ptr %a, double ${i}) ; borrowed scalar union`); diff --git a/packages/compiler/src/backend/loop-polls.ts b/packages/compiler/src/backend/loop-polls.ts index 14c2b049c..c582a4974 100644 --- a/packages/compiler/src/backend/loop-polls.ts +++ b/packages/compiler/src/backend/loop-polls.ts @@ -106,3 +106,34 @@ export function loopTestsEveryIteration(loop: Loop, tested: (call: IrExpr) => bo } return false; } + +/** True when the body of `loop`, outside its nested loops and closures, + * reads or writes a typed-array element: the loops whose cancellation poll + * the emitter moves between blocks of iterations, so that LLVM can + * vectorize them. Other loops keep the budget in their body, which costs + * less in a loop of a few iterations. */ +export function loopAccessesTypedArrays(loop: Loop): boolean { + const inExpr = (e: IrExpr): boolean => { + if (e.kind === "bytesIntrinsic" && e.method === "get") return true; + let found = false; + everyExprChild( + e, + (child) => !(found = inExpr(child)), + (child) => !(found = inStmt(child)), + ); + return found; + }; + const inStmt = (s: IrStmt): boolean => { + if (s.kind === "bytesSet") return true; + if (s.kind === "while" || s.kind === "doWhile" || s.kind === "for" || s.kind === "forOf") + return false; + let found = false; + everyStmtChild( + s, + (child) => !(found = inExpr(child)), + (child) => !(found = inStmt(child)), + ); + return found; + }; + return loop.body.some(inStmt); +} diff --git a/packages/compiler/src/backend/runtime-abi.ts b/packages/compiler/src/backend/runtime-abi.ts index 7ac0877eb..fe13879a2 100644 --- a/packages/compiler/src/backend/runtime-abi.ts +++ b/packages/compiler/src/backend/runtime-abi.ts @@ -2,8 +2,8 @@ * of the link contract: a program object keeps this symbol undefined and the * matching runtime defines it, so incompatible manual links fail before the * program can start. */ -export const RUNTIME_ABI_VERSION = 44 as const; -export const RUNTIME_ABI_MARKER = "scr_runtime_abi_v44" as const; +export const RUNTIME_ABI_VERSION = 45 as const; +export const RUNTIME_ABI_MARKER = "scr_runtime_abi_v45" as const; /** Object consumption is useful today, but the complete scr_* surface may * still change before 1.0. The versioned marker prevents accidental mixing; diff --git a/packages/compiler/src/backend/runtime-pack-targets.test.ts b/packages/compiler/src/backend/runtime-pack-targets.test.ts index 650e7c543..492b71217 100644 --- a/packages/compiler/src/backend/runtime-pack-targets.test.ts +++ b/packages/compiler/src/backend/runtime-pack-targets.test.ts @@ -52,7 +52,7 @@ async function fixture(target: NativeTargetSpec, workers = false) { object_format: target.objectFormat, minimum_os: target.minimumOs, }, - runtime_abi: { version: 44, marker: "scr_runtime_abi_v44" }, + runtime_abi: { version: 45, marker: "scr_runtime_abi_v45" }, compiler: { command: "fixture", identity: "fixture", target: target.llvmTriple }, macros: { executable: [], excluded: ["SCR_LIB"], sanitizer: "external-toolchain-required" }, flavors: { diff --git a/packages/compiler/src/backend/runtime-pack.test.ts b/packages/compiler/src/backend/runtime-pack.test.ts index 443c22d9e..fdb0ba29e 100644 --- a/packages/compiler/src/backend/runtime-pack.test.ts +++ b/packages/compiler/src/backend/runtime-pack.test.ts @@ -129,7 +129,7 @@ async function fixture(target: NativeTargetSpec = MACOS_ARM64_TARGET) { object_format: target.objectFormat, minimum_os: target.minimumOs, }, - runtime_abi: { version: 44, marker: "scr_runtime_abi_v44" }, + runtime_abi: { version: 45, marker: "scr_runtime_abi_v45" }, compiler: { command: "clang", identity: "fixture clang", diff --git a/packages/compiler/src/cli/command.ts b/packages/compiler/src/cli/command.ts index da3a09cf0..93d81f7e7 100644 --- a/packages/compiler/src/cli/command.ts +++ b/packages/compiler/src/cli/command.ts @@ -19,7 +19,12 @@ import { } from "../node-islands/policy.js"; import { TIER_MODES, type TierMode } from "../frontend/tier-analysis.js"; import { buildEnvelope } from "../diagnostics/envelope.js"; -import { renderConfigWarnings, renderWarnings } from "../diagnostics/render.js"; +import { + renderConfigWarnings, + renderTypePatchWarnings, + renderWarnings, +} from "../diagnostics/render.js"; +import { isTypePatchCode } from "../diagnostics/diagnostic.js"; import { isPerformanceCode, renderPerformanceWarnings } from "../diagnostics/dynamic-tier.js"; import { isNativeOptimization } from "../backend/optimization.js"; import { @@ -645,20 +650,30 @@ async function main(args: string[], host: CliHost): Promise { // keeps stderr for the program's own output. const color = process.stderr.isTTY ?? false; const config = warnings.filter((d) => d.code === "SC8004"); - const divergences = warnings.filter((d) => !isPerformanceCode(d.code) && d.code !== "SC8004"); + const patches = warnings.filter((d) => isTypePatchCode(d.code)); + const divergences = warnings.filter( + (d) => !isPerformanceCode(d.code) && d.code !== "SC8004" && !isTypePatchCode(d.code), + ); const performance = warnings.filter((d) => isPerformanceCode(d.code)); if (config.length > 0) process.stderr.write( renderConfigWarnings(config, result.sourceTexts ?? new Map(), { color }) + "\n", ); - if (divergences.length > 0) { + if (patches.length > 0) { if (config.length > 0) process.stderr.write("\n"); + process.stderr.write( + renderTypePatchWarnings(patches, result.sourceTexts ?? new Map(), { color }) + "\n", + ); + } + if (divergences.length > 0) { + if (config.length > 0 || patches.length > 0) process.stderr.write("\n"); process.stderr.write( renderWarnings(divergences, result.sourceTexts ?? new Map(), { color }) + "\n", ); } if (performance.length > 0) { - if (divergences.length > 0 || config.length > 0) process.stderr.write("\n"); + if (divergences.length > 0 || config.length > 0 || patches.length > 0) + process.stderr.write("\n"); process.stderr.write(renderPerformanceWarnings(performance) + "\n"); } } diff --git a/packages/compiler/src/diagnostics/diagnostic.ts b/packages/compiler/src/diagnostics/diagnostic.ts index 9c6308ec4..f272409a6 100644 --- a/packages/compiler/src/diagnostics/diagnostic.ts +++ b/packages/compiler/src/diagnostics/diagnostic.ts @@ -71,9 +71,11 @@ * required to be native (SC8003) or every island refused * (SC8002) while outside the native subset, and the R2 * object-model constructs in user JavaScript (SC8101–SC8105, - * behind --strict-user-js); and programs that call into - * Node.js where the build finds no usable libnode pack - * (SC8005), or cannot collect the modules a call loads (SC8006) + * behind --strict-user-js); programs that call into Node.js + * where the build finds no usable libnode pack (SC8005), or + * cannot collect the modules a call loads (SC8006); and type + * patches from `scriptc.typePatches` that did not apply as + * written (SC8030–SC8033, warnings) * SC9xxx internal compiler errors (still source-anchored) */ import type { SrcLoc } from "../ir/ir.js"; @@ -289,6 +291,25 @@ export function nodeIslandsOffModuleDiag( }; } +/** SC8030–SC8033 — a type patch (`scriptc.typePatches`) that did not + * apply as written: no range covers the installed copy (SC8030), its + * selector matches nothing (SC8031), the declaration already has the type + * (SC8032), or the selector is ambiguous or not patchable (SC8033). + * Warnings: the build goes on with the declarations it has. */ +export function typePatchDiag( + code: "SC8030" | "SC8031" | "SC8032" | "SC8033", + message: string, + hint: string, + loc: SrcLoc, +): ScrDiagnostic { + return { code, message, loc, hint }; +} + +/** True for the type patch warnings (SC8030–SC8033). */ +export function isTypePatchCode(code: string): boolean { + return /^SC803[0-3]$/.test(code); +} + /** SC8003 — a package marked "native" is outside the native subset. */ export function nativePackageDiag(pkg: string, why: string, loc: SrcLoc): ScrDiagnostic { return { diff --git a/packages/compiler/src/diagnostics/envelope.ts b/packages/compiler/src/diagnostics/envelope.ts index ad2e31507..bdb545ff3 100644 --- a/packages/compiler/src/diagnostics/envelope.ts +++ b/packages/compiler/src/diagnostics/envelope.ts @@ -15,7 +15,7 @@ * version; renaming or removing a field, or changing a field's meaning, * bumps it. */ import { readFileSync } from "node:fs"; -import type { ScrDiagnostic } from "./diagnostic.js"; +import { isTypePatchCode, type ScrDiagnostic } from "./diagnostic.js"; import { isPerformanceCode } from "./dynamic-tier.js"; import type { WasmModuleReport } from "../wasm/discover.js"; @@ -217,6 +217,7 @@ export function categoryOf(diag: ScrDiagnostic): DiagnosticCategory { if (code === "SC0002" || code === "SC0004") return "environment"; if (/^SC[35]\d{3}$/.test(code)) return "environment"; if (code === "SC8004" || code === "SC8005" || code === "SC8012") return "environment"; + if (isTypePatchCode(code)) return "environment"; if (/^SC9\d{3}$/.test(code)) return "internal"; return "unsupported"; } @@ -335,7 +336,10 @@ export function locateDiagnostic( code: diag.code, category: categoryOf(diag), severity: - isDivergenceCode(diag.code) || isPerformanceCode(diag.code) || diag.code === "SC8004" + isDivergenceCode(diag.code) || + isPerformanceCode(diag.code) || + diag.code === "SC8004" || + isTypePatchCode(diag.code) ? "warning" : "error", scope, diff --git a/packages/compiler/src/diagnostics/render.ts b/packages/compiler/src/diagnostics/render.ts index 16fe3f0bf..a370eb3b1 100644 --- a/packages/compiler/src/diagnostics/render.ts +++ b/packages/compiler/src/diagnostics/render.ts @@ -4,7 +4,7 @@ * longer than FRAME_MAX_WIDTH (minified sources) shows a window of that * width around the diagnostic's column, with "…" where text is cut. */ -import type { ScrDiagnostic } from "./diagnostic.js"; +import { isTypePatchCode, type ScrDiagnostic } from "./diagnostic.js"; export interface RenderOptions { color?: boolean; @@ -157,7 +157,20 @@ export function renderDiagnostics( * settings in the project's scriptc configuration (SC8004), which builds do * not use. */ export function isWarningCode(code: string): boolean { - return /^SC6\d{3}$/.test(code) || code === "SC8004"; + return /^SC6\d{3}$/.test(code) || code === "SC8004" || isTypePatchCode(code); +} + +/** A build's type patch warnings (SC8030–SC8033). */ +export function renderTypePatchWarnings( + diags: readonly ScrDiagnostic[], + sourceTextByFile: Map, + opts: RenderOptions = {}, +): string { + const n = diags.length; + return ( + renderDiagnostics([...diags], sourceTextByFile, opts) + + `\n\n${n} warning${n === 1 ? "" : "s"}: ${n === 1 ? "this type patch" : "these type patches"} did not apply as written; the build uses the package's own declarations there.` + ); } /** A build's SC8004 warnings: the unknown scriptc settings it ignored. */ diff --git a/packages/compiler/src/frontend/lowering/array-values.ts b/packages/compiler/src/frontend/lowering/array-values.ts index 6e4a45f86..ed52b8da5 100644 --- a/packages/compiler/src/frontend/lowering/array-values.ts +++ b/packages/compiler/src/frontend/lowering/array-values.ts @@ -58,7 +58,33 @@ export function arrayNumberRead(arr: IrExpr, index: IrExpr, loc: SrcLoc): IrExpr }; } -/** Read one stabilized receiver/index pair. Both a hole and present undefined yield undefined. */ +/** A foreign element read into a typed result: it never fits, so the read + * throws the checked conversion's TypeError (array.foreignThrow). */ +export function foreignElementThrow( + lowerer: Lowerer, + arr: IrExpr, + index: IrExpr, + type: IrType, + loc: SrcLoc, +): IrExpr { + return { + kind: "libCall", + fn: "array.foreignThrow", + args: [arr, index, { kind: "strLit", value: lowerer.fmt(type), type: STRING, loc }], + type, + loc, + }; +} + +/** Arrays of this element type can hold elements outside it, stored + * through dynamic views (Lowerer.foreignElementKeys). */ +export function elementMayBeForeign(lowerer: Lowerer, elem: IrType): boolean { + const keys = lowerer.foreignElementKeys; + return keys.size > 0 && elem.kind !== "dyn" && (keys.has("*") || keys.has(typeKey(elem))); +} + +/** Read one stabilized receiver/index pair. Both a hole and present undefined yield undefined; + * a foreign element (state 3) takes the checked conversion into the typed result. */ export function arrayValueRead( lowerer: Lowerer, arr: IrExpr, @@ -68,8 +94,25 @@ export function arrayValueRead( ): IrExpr { const type = arrayValueType(lowerer, elem); const read: IrExpr = { kind: "arrayGet", arr, index, type: elem, loc }; - const missing: IrExpr = + let missing: IrExpr = type.kind === "dyn" ? dynUndefinedExpr(loc) : lowerer.wrappedUndefined(type, loc)!; + if (elementMayBeForeign(lowerer, elem)) { + missing = { + kind: "ternary", + cond: { + kind: "bin", + op: "===", + left: { kind: "arrayState", arr, index, type: F64, loc }, + right: { kind: "numLit", value: 3, type: F64, loc }, + type: BOOL, + loc, + }, + then: foreignElementThrow(lowerer, arr, index, type, loc), + else_: missing, + type, + loc, + }; + } return { kind: "ternary", cond: { @@ -684,6 +727,9 @@ export function tryLowerNumericIndexRead( operand.callee !== lowerer.arrHofHelpers.get(`idxOr:${typeKey(F64)}`) ) return null; + // ToNumber of a foreign element differs from its own value under `+`, + // comparisons and ToBoolean. + if (elementMayBeForeign(lowerer, F64)) return null; const [arr, index] = operand.args; if ( operand.args.length !== 2 || @@ -777,6 +823,8 @@ export function tryFuseIndexRead( if (arr?.type.kind !== "array" || index?.type.kind !== "f64") return null; const elem = arr.type.elem; if (elem.kind === "union" || elem.kind === "void") return null; + // A foreign element is neither the present value nor the missing one. + if (elementMayBeForeign(lowerer, elem)) return null; if (operand.callee !== lowerer.arrHofHelpers.get(`idxOr:${typeKey(elem)}`)) return null; if (arr.kind !== "varRef" || (index.kind !== "varRef" && index.kind !== "numLit")) return null; const loc = operand.loc; diff --git a/packages/compiler/src/frontend/lowering/coercions/state.ts b/packages/compiler/src/frontend/lowering/coercions/state.ts index ad1d7c6ea..db9bb53ad 100644 --- a/packages/compiler/src/frontend/lowering/coercions/state.ts +++ b/packages/compiler/src/frontend/lowering/coercions/state.ts @@ -12,7 +12,10 @@ export class CoercionState { // Provenance for checked extraction; consumers must not infer it from names. readonly checkedNarrows = new Set(); // The extracted arm of each checked (non-defaulting) extraction helper. - readonly checkedNarrowArms = new Map(); + readonly checkedNarrowArms = new Map< + string, + { unionId: string; tag: number; target: IrType; message?: string } + >(); readonly retags = new Map(); readonly narrows = new Map(); } diff --git a/packages/compiler/src/frontend/lowering/coercions/unions.ts b/packages/compiler/src/frontend/lowering/coercions/unions.ts index 434e687a5..53c87e429 100644 --- a/packages/compiler/src/frontend/lowering/coercions/unions.ts +++ b/packages/compiler/src/frontend/lowering/coercions/unions.ts @@ -394,18 +394,21 @@ export function unionRetagHelper( * the catchable TypeError — divergence 38's lying-assertion stance (an * unchecked unionNarrow would misread the payload where JS lets the * impossible value flow on). Null when the target isn't a non-unit arm - * of the union — those uses keep their erasure/fences. */ + * of the union — those uses keep their erasure/fences. `message` replaces + * the failure message (the TypeError of the operation consuming the + * value, such as a call's "f is not a function"). */ export function narrowedArmHelper( lowerer: Lowerer, fromId: string, target: IrType, loc: SrcLoc, + message?: string, ): string | null { const from = lowerer.unions.get(fromId); if (!from || isUnitType(target)) return null; const tag = lowerer.armTag(fromId, target); if (tag < 0) return null; - const key = `${fromId}:${tag}`; + const key = message === undefined ? `${fromId}:${tag}` : `${fromId}:${tag}:${message}`; const existing = lowerer.coercions.narrows.get(key); if (existing) { lowerer.coercions.checkedNarrows.add(existing); @@ -414,9 +417,22 @@ export function narrowedArmHelper( const name = keyedHelperName(lowerer, "%union.narrow", key); lowerer.coercions.narrows.set(key, name); lowerer.coercions.checkedNarrows.add(name); - const fn = buildUnionNarrow(name, from, target, loc, (type) => lowerer.fmt(type)); + const fn = buildUnionNarrow( + name, + from, + target, + loc, + (type) => lowerer.fmt(type), + null, + message ?? null, + ); if (!fn) throw new InternalCompilerError("lowerer bug: invalid checked union extraction"); - lowerer.coercions.checkedNarrowArms.set(name, { unionId: fromId, tag, target: from.arms[tag]! }); + lowerer.coercions.checkedNarrowArms.set(name, { + unionId: fromId, + tag, + target: from.arms[tag]!, + ...(message === undefined ? {} : { message }), + }); lowerer.liftedFns.push(fn); return name; } @@ -458,7 +474,8 @@ export function checkedNarrowCall( let failure: IrExpr | null = null; for (let tag = from.arms.length - 1; tag >= 0; tag--) { if (tag === arm.tag) continue; - const message = unionNarrowFailureMessage(from.arms[tag]!, (t) => lowerer.fmt(t)); + const message = + arm.message ?? unionNarrowFailureMessage(from.arms[tag]!, (t) => lowerer.fmt(t)); const thrown = nodeThrowExpr(1, "", message, type, loc); failure = failure === null diff --git a/packages/compiler/src/frontend/lowering/containers/array-callbacks.ts b/packages/compiler/src/frontend/lowering/containers/array-callbacks.ts index 46459fadb..2c4d5d9b4 100644 --- a/packages/compiler/src/frontend/lowering/containers/array-callbacks.ts +++ b/packages/compiler/src/frontend/lowering/containers/array-callbacks.ts @@ -42,6 +42,8 @@ import { arrayValueStore, arrayValueType, currentArrayIndexPresent, + elementMayBeForeign, + foreignElementThrow, primitiveElement, } from "../array-values.js"; import { typeKey } from "../../type-mapper.js"; @@ -2531,6 +2533,9 @@ export function lowerArraySortCall( loc, native, keyed && { order, key: reads !== null, keys: keyed.length }, + elementMayBeForeign(lowerer, elem) + ? (arr, index) => foreignElementThrow(lowerer, arr, index, elem, loc) + : null, ), ); } diff --git a/packages/compiler/src/frontend/lowering/containers/array-concat.ts b/packages/compiler/src/frontend/lowering/containers/array-concat.ts index a7ce0a106..440cbe9e2 100644 --- a/packages/compiler/src/frontend/lowering/containers/array-concat.ts +++ b/packages/compiler/src/frontend/lowering/containers/array-concat.ts @@ -1,5 +1,6 @@ import { keyedHelperName } from "../stable-names.js"; import { identityPreservingWidening } from "../coercions/identity.js"; +import { elementMayBeForeign, foreignElementThrow } from "../array-values.js"; import { countedFor, numLit, varRef } from "../../../ir/build.js"; import { BOOL, @@ -131,7 +132,33 @@ export function arrayConcatHelper( loc, }, then: [{ kind: "arraySetUndefined", arr: out, index: destination, loc }], - else_: null, + else_: elementMayBeForeign(lowerer, sourceElem) + ? [ + { + // A foreign element takes the checked conversion's TypeError. + kind: "if", + cond: { + kind: "bin", + op: "===", + left: { kind: "arrayState", arr: source, index, type: F64, loc }, + right: numLit(3, loc), + type: BOOL, + loc, + }, + then: [ + { + kind: "arraySet", + arr: out, + index: destination, + value: foreignElementThrow(lowerer, source, index, elem, loc), + loc, + }, + ], + else_: null, + loc, + }, + ] + : null, loc, }, ], diff --git a/packages/compiler/src/frontend/lowering/containers/array-construction.ts b/packages/compiler/src/frontend/lowering/containers/array-construction.ts index 5029abc36..702837e40 100644 --- a/packages/compiler/src/frontend/lowering/containers/array-construction.ts +++ b/packages/compiler/src/frontend/lowering/containers/array-construction.ts @@ -1081,9 +1081,7 @@ export function lowerArrayFromCall( if ((args.length === 1 || args.length === 2) && !ts.isObjectLiteralExpression(args[0]!)) { const source = lowerer.lowerExpr(args[0]!); let src = - source.type.kind === "string" && args.length === 2 - ? strCharsCall(lowerer, source, loc) - : source; + source.type.kind === "string" && args.length === 2 ? strCharsCall(source, loc) : source; if (src.type.kind === "array") { const arrT = src.type; if (args.length === 1) { @@ -1171,7 +1169,7 @@ export function lowerArrayFromCall( } // String iteration advances by code point; the same helper serves // `[...s]` and both Array.from(s) forms. - if (src.type.kind === "string" && args.length === 1) return strCharsCall(lowerer, src, loc); + if (src.type.kind === "string" && args.length === 1) return strCharsCall(src, loc); if (src.type.kind === "dyn" && args.length === 1) { return lowerCheckedArrayFrom(lowerer, src, loc); } @@ -1318,112 +1316,9 @@ function buildArrayFromArrayFn( } /** `Array.from(s)` / `[...s]` on a STRING: the code-point split into a - * fresh string[], through one interned helper per module. */ -export function strCharsCall(lowerer: Lowerer, src: IrExpr, loc: SrcLoc): IrExpr { - const key = "strChars"; - let helper = lowerer.arrHofHelpers.get(key); - if (!helper) { - helper = "%str.chars"; - lowerer.arrHofHelpers.set(key, helper); - lowerer.liftedFns.push(buildStrCharsFn(helper, loc)); - } - return { kind: "call", callee: helper, args: [src], type: arrayOf(STRING), loc }; -} - -/** The code-point split, from existing IR nodes — the string for-of - * desugar's UTF-16 cursor as a function: - * - * out = []; - * i = 0; - * while (i < s.length) { ch = cpAt(s, i); i += ch.length; out.push(ch); } - * return out; - */ -function buildStrCharsFn(name: string, loc: SrcLoc): IrFunction { - const outT = arrayOf(STRING); - - const sLen = (recv: IrExpr): IrExpr => ({ - kind: "strIntrinsic", - method: "length", - receiver: recv, - args: [], - type: F64, - loc, - }); - const body: IrStmt[] = [ - { - kind: "varDecl", - localId: "out.0", - init: { kind: "arrayLit", elems: [], type: outT, loc }, - loc, - }, - { kind: "varDecl", localId: "i.0", init: { kind: "numLit", value: 0, type: F64, loc }, loc }, - { - kind: "while", - cond: { - kind: "bin", - op: "<", - left: varRef("i.0", F64, loc), - right: sLen(varRef("s.0", STRING, loc)), - type: BOOL, - loc, - }, - body: [ - { - kind: "varDecl", - localId: "ch.0", - init: { - kind: "strIntrinsic", - method: "cpAt", - receiver: varRef("s.0", STRING, loc), - args: [varRef("i.0", F64, loc)], - type: STRING, - loc, - }, - loc, - }, - { - kind: "assign", - localId: "i.0", - value: { - kind: "bin", - op: "+", - left: varRef("i.0", F64, loc), - right: sLen(varRef("ch.0", STRING, loc)), - type: F64, - loc, - }, - loc, - }, - { - kind: "exprStmt", - expr: { - kind: "arrIntrinsic", - method: "push", - receiver: varRef("out.0", outT, loc), - args: [varRef("ch.0", STRING, loc)], - type: F64, - loc, - }, - loc, - }, - ], - loc, - }, - { kind: "return", value: varRef("out.0", outT, loc), loc }, - ]; - return { - name, - params: [{ localId: "s.0", name: "s", type: STRING }], - returnType: outT, - locals: [ - { id: "s.0", name: "s", type: STRING, mutable: true }, - { id: "out.0", name: "out", type: outT, mutable: false }, - { id: "i.0", name: "i", type: F64, mutable: true }, - { id: "ch.0", name: "ch", type: STRING, mutable: false }, - ], - body, - loc, - }; + * fresh string[] (scr_str_code_points). */ +export function strCharsCall(src: IrExpr, loc: SrcLoc): IrExpr { + return { kind: "libCall", fn: "str.codePoints", args: [src], type: arrayOf(STRING), loc }; } /** The lowered `length` value of the one-property source literal, or null diff --git a/packages/compiler/src/frontend/lowering/containers/array-methods.ts b/packages/compiler/src/frontend/lowering/containers/array-methods.ts index f406e57eb..3b16e714f 100644 --- a/packages/compiler/src/frontend/lowering/containers/array-methods.ts +++ b/packages/compiler/src/frontend/lowering/containers/array-methods.ts @@ -1886,17 +1886,27 @@ function arraySearchHelper( }, loc, }; - const visitPresent: IrStmt[] = [ - { kind: "varDecl", localId: "value.0", init: value, loc }, - { - kind: "if", - cond: match, - then: [{ kind: "return", value: method === "includes" ? boolLit(true, loc) : indexRef, loc }], - else_: null, - loc, - }, - advance, - ]; + const found: IrStmt = { + kind: "if", + cond: directNeedle ? equal : match, + then: [{ kind: "return", value: method === "includes" ? boolLit(true, loc) : indexRef, loc }], + else_: null, + loc, + }; + // A direct needle reads only value slots: an undefined or foreign + // element written through a dynamic view has no element-typed payload. + const visitPresent: IrStmt[] = directNeedle + ? [ + { + kind: "if", + cond: { kind: "bin", op: "===", left: stateRef, right: numLit(1, loc), type: BOOL, loc }, + then: [{ kind: "varDecl", localId: "value.0", init: value, loc }, found], + else_: null, + loc, + }, + advance, + ] + : [{ kind: "varDecl", localId: "value.0", init: value, loc }, found, advance]; const skipHole: IrStmt[] = method === "lastIndexOf" ? [advance] diff --git a/packages/compiler/src/frontend/lowering/containers/array-pipelines.ts b/packages/compiler/src/frontend/lowering/containers/array-pipelines.ts index 197af2a9e..6307d871f 100644 --- a/packages/compiler/src/frontend/lowering/containers/array-pipelines.ts +++ b/packages/compiler/src/frontend/lowering/containers/array-pipelines.ts @@ -34,6 +34,7 @@ import { arrayValueRead, arrayValueStore, arrayValueType, + elementMayBeForeign, } from "../array-values.js"; import type { Lowerer } from "../lowerer.js"; @@ -156,7 +157,8 @@ const PURE_STRING = new Set([ "toUpperCase", "isWellFormed", "toWellFormed", - "cpAt", + "byteLength", + "cpAtByte", ]); function primitive(t: IrType): boolean { @@ -1643,7 +1645,9 @@ class FusedLoop { ]; } const valueT = arrayValueType(this.lowerer, arrT.elem); - if (stage.fast) { + // A foreign element is neither the plain element nor undefined: the + // ordinary read below converts it. + if (stage.fast && !elementMayBeForeign(this.lowerer, arrT.elem)) { // The split visit: a value goes on as the plain element, a present // undefined as the undefined arm. const missing = this.lowerer.wrappedUndefined(valueT, loc); diff --git a/packages/compiler/src/frontend/lowering/containers/collection-methods.ts b/packages/compiler/src/frontend/lowering/containers/collection-methods.ts index f18291a63..bfaca6f96 100644 --- a/packages/compiler/src/frontend/lowering/containers/collection-methods.ts +++ b/packages/compiler/src/frontend/lowering/containers/collection-methods.ts @@ -1135,7 +1135,7 @@ function setFromSeedValue( if (source.type.kind === "set" && typeEquals(source.type, setT)) { return { kind: "setIntrinsic", method: "clone", receiver: source, args: [], type: setT, loc }; } else if (source.type.kind === "string" && setT.elem.kind === "string") { - seed = strCharsCall(lowerer, source, loc); + seed = strCharsCall(source, loc); } else if (source.type.kind === "array" && typeEquals(source.type.elem, setT.elem)) { seed = source; } else if ( diff --git a/packages/compiler/src/frontend/lowering/expressions/array-literals.ts b/packages/compiler/src/frontend/lowering/expressions/array-literals.ts index 903f0884d..77c16594f 100644 --- a/packages/compiler/src/frontend/lowering/expressions/array-literals.ts +++ b/packages/compiler/src/frontend/lowering/expressions/array-literals.ts @@ -504,7 +504,7 @@ export function lowerArrayLiteral( // interned helper as Array.from(s); the result rides the array // machinery below like any string[] source. if (src.type.kind === "string") { - src = strCharsCall(lowerer, src, locOf(el)); + src = strCharsCall(src, locOf(el)); } // `[...typedArray]`: represented typed arrays are dense numeric // iterables, so drain their elements into the fresh number[] that diff --git a/packages/compiler/src/frontend/lowering/expressions/element-access.ts b/packages/compiler/src/frontend/lowering/expressions/element-access.ts index a2f2ebdb6..59b6574e6 100644 --- a/packages/compiler/src/frontend/lowering/expressions/element-access.ts +++ b/packages/compiler/src/frontend/lowering/expressions/element-access.ts @@ -59,6 +59,7 @@ import { arrayValueRead, arrayValueStore, arrayValueType, + elementMayBeForeign, lowerSafeBytesRead, lowerSafeIndexRead, primitiveElement, @@ -1623,11 +1624,14 @@ export function lowerElementCompoundParts( const indexRef = (): IrExpr => varRef(indexLocal.id, F64, loc); const oldValue: IrExpr = receiver.type.kind === "array" - ? receiver.type.elem.kind === "f64" + ? receiver.type.elem.kind === "f64" && + !(op === "+" && elementMayBeForeign(lowerer, receiver.type.elem)) ? // Every numeric operator applies ToNumber to the old value, so a // missing element is NaN without materializing `number | undefined`. arrayNumberRead(receiverRef(), indexRef(), locOf(target)) - : receiver.type.elem.kind === "string" && op === "+" + : receiver.type.elem.kind === "string" && + op === "+" && + !elementMayBeForeign(lowerer, receiver.type.elem) ? arrayStringOperand(receiverRef(), indexRef(), locOf(target)) : arrayValueRead(lowerer, receiverRef(), indexRef(), receiver.type.elem, locOf(target)) : { diff --git a/packages/compiler/src/frontend/lowering/expressions/narrowing.ts b/packages/compiler/src/frontend/lowering/expressions/narrowing.ts index 01cab9901..9f0559672 100644 --- a/packages/compiler/src/frontend/lowering/expressions/narrowing.ts +++ b/packages/compiler/src/frontend/lowering/expressions/narrowing.ts @@ -875,8 +875,15 @@ function checkedStaleArm( /** V8's spelling of a callee in "is not a function": a string-literal * element key reads as a property name (`o["a"]` is `o.a`), other keys - * keep their brackets. */ + * keep their brackets, parentheses and non-null assertions disappear, and + * a call result reads `f(...)`. */ export function v8CalleeText(node: ts.Node): string { + // `!` is type-only syntax: the running JavaScript has no trace of it. + if (ts.isNonNullExpression(node) || ts.isParenthesizedExpression(node)) + return v8CalleeText(node.expression); + // A call result prints with its arguments elided. + if (ts.isCallExpression(node) && !node.questionDotToken) + return `${v8CalleeText(node.expression)}(...)`; if (ts.isElementAccessExpression(node)) { const key = node.argumentExpression; const object = v8CalleeText(node.expression); diff --git a/packages/compiler/src/frontend/lowering/expressions/nullish-assignment.ts b/packages/compiler/src/frontend/lowering/expressions/nullish-assignment.ts index 9664f591d..8a25eb124 100644 --- a/packages/compiler/src/frontend/lowering/expressions/nullish-assignment.ts +++ b/packages/compiler/src/frontend/lowering/expressions/nullish-assignment.ts @@ -14,7 +14,12 @@ import { import { varRef } from "../../../ir/build.js"; import { locOf } from "../../program.js"; import type { Lowerer } from "../lowerer.js"; -import { arrayNumberRead, arrayValueRead, arrayValueStore } from "../array-values.js"; +import { + arrayNumberRead, + arrayValueRead, + arrayValueStore, + elementMayBeForeign, +} from "../array-values.js"; import { fenceNodeModuleMutation } from "../lower-node-module.js"; import { globalSymbolKey } from "./global-symbols.js"; import { tryLowerExpression } from "./try-lower-expression.js"; @@ -185,7 +190,12 @@ export function lowerShortCircuitAssignment(lowerer: Lowerer, expr: ts.BinaryExp const element = storage; write = (value) => arrayValueStore(lowerer, receiver, key, value, element, loc); const elementLoc = locOf(target); - if (nullish && NON_NULLISH_ELEMENTS.has(element.kind)) { + // A foreign element can be any value, nullish or not. + if ( + nullish && + NON_NULLISH_ELEMENTS.has(element.kind) && + !elementMayBeForeign(lowerer, element) + ) { // Only a missing element is nullish: keep the present value or // store the right side, without the optional element value. rhs = lowerer.lowerExprExpecting(expr.right, element); diff --git a/packages/compiler/src/frontend/lowering/generator-delegation.ts b/packages/compiler/src/frontend/lowering/generator-delegation.ts index a606c177e..81ecc8353 100644 --- a/packages/compiler/src/frontend/lowering/generator-delegation.ts +++ b/packages/compiler/src/frontend/lowering/generator-delegation.ts @@ -577,10 +577,11 @@ export function lowerBuiltinDelegation( loc, }); } else if (sourceT.kind === "string") { + // A byte cursor over the WTF-8 data, as for-of over a string walks. const ch = lowerer.declareHiddenLocal("%delegateElement", STRING); const length = (receiver: IrExpr): IrExpr => ({ kind: "strIntrinsic", - method: "length", + method: "byteLength", receiver, args: [], type: F64, @@ -595,7 +596,7 @@ export function lowerBuiltinDelegation( localId: ch.id, init: { kind: "strIntrinsic", - method: "cpAt", + method: "cpAtByte", receiver: src(), args: [at()], type: STRING, diff --git a/packages/compiler/src/frontend/lowering/iterator-adapters.ts b/packages/compiler/src/frontend/lowering/iterator-adapters.ts index 4909e7f45..78fcff84d 100644 --- a/packages/compiler/src/frontend/lowering/iterator-adapters.ts +++ b/packages/compiler/src/frontend/lowering/iterator-adapters.ts @@ -743,7 +743,7 @@ export function lowerStringIteratorValue( left: index, right: { kind: "strIntrinsic", - method: "length", + method: "byteLength", receiver: text, args: [], type: F64, @@ -758,7 +758,7 @@ export function lowerStringIteratorValue( localId: "point.0", init: { kind: "strIntrinsic", - method: "cpAt", + method: "cpAtByte", receiver: text, args: [index], type: STRING, @@ -775,7 +775,7 @@ export function lowerStringIteratorValue( left: index, right: { kind: "strIntrinsic", - method: "length", + method: "byteLength", receiver: point, args: [], type: F64, diff --git a/packages/compiler/src/frontend/lowering/lower-array-sort.ts b/packages/compiler/src/frontend/lowering/lower-array-sort.ts index 5521d30ef..62507ddcd 100644 --- a/packages/compiler/src/frontend/lowering/lower-array-sort.ts +++ b/packages/compiler/src/frontend/lowering/lower-array-sort.ts @@ -926,6 +926,10 @@ export function buildArraySortFn( loc: SrcLoc, native = false, keyed: { order: number; key: boolean; keys: number } | null = null, + /** The throw for a foreign element (SCR_ARR_FOREIGN) at `index` of + * `arr`, which the comparator's element type cannot hold; for element + * types whose arrays can hold one. */ + checkForeign: ((arr: IrExpr, index: IrExpr) => IrExpr) | null = null, ): IrFunction { const b = new SortIr(loc); const arrT = arrayOf(elem), @@ -1035,7 +1039,21 @@ export function buildArraySortFn( b.branch( b.eq(state, b.num(1)), [compactValue()], - [b.branch(b.eq(state, b.num(2)), [b.step(undefinedCount)])], + [ + b.branch( + b.eq(state, b.num(2)), + [b.step(undefinedCount)], + checkForeign + ? [ + // A foreign element does not fit the comparator's element + // type: the checked conversion's TypeError. + b.branch(b.eq(state, b.num(3)), [ + { kind: "exprStmt", expr: checkForeign(src, index), loc }, + ]), + ] + : null, + ), + ], ), ]), // Sort only the compacted prefix. The tail is overwritten with undefined diff --git a/packages/compiler/src/frontend/lowering/lower-exprs.ts b/packages/compiler/src/frontend/lowering/lower-exprs.ts index 20d05f3b7..006975085 100644 --- a/packages/compiler/src/frontend/lowering/lower-exprs.ts +++ b/packages/compiler/src/frontend/lowering/lower-exprs.ts @@ -89,7 +89,7 @@ import { classSymbolKeyOf } from "./symbol-fields.js"; import { lowerTypeofExpr } from "./expressions/typeof.js"; import { lowerIdentifierExpr } from "./expressions/identifiers.js"; import { lowerFieldCompoundValue, lowerIncDec } from "./expressions/updates.js"; -import { lowerTernary } from "./expressions/narrowing.js"; +import { lowerTernary, v8CalleeText } from "./expressions/narrowing.js"; import { classFieldTarget, lowerPropertyAccessExpr, @@ -433,7 +433,12 @@ function lowerExprInner(lowerer: Lowerer, expr: ts.Expression): IrExpr { const operand = assertionOperand(lowerer, expr, inner, target ?? null, loc); if (operand) return operand; if (target && target.kind !== "union" && !typeEquals(target, inner.type)) { - const helper = lowerer.narrowedArmHelper(inner.type.unionId, target, loc); + const helper = lowerer.narrowedArmHelper( + inner.type.unionId, + target, + loc, + assertedCalleeMessage(expr, target), + ); if (helper) { return lowerer.checkedNarrowCall(helper, inner, target, loc); } @@ -1160,6 +1165,19 @@ const NUMERIC_COMPOUNDS = new Set([ ts.SyntaxKind.GreaterThanGreaterThanGreaterThanEqualsToken, ]); +/** The TypeError Node raises when an asserted callee (`t[k]!(…)`, `o.f!()`) + * holds no function: `!` is erased, so the call itself throws " is + * not a function" in V8's spelling of the callee. */ +function assertedCalleeMessage(expr: ts.NonNullExpression, target: IrType): string | undefined { + if (target.kind !== "func") return undefined; + let child: ts.Expression = expr; + while (ts.isParenthesizedExpression(child.parent)) child = child.parent; + const call = child.parent; + if (!ts.isCallExpression(call) || call.expression !== child || call.questionDotToken) + return undefined; + return `${v8CalleeText(expr.expression)} is not a function`; +} + /** Whether the use of `t[i]!` tells undefined from NaN: a converting use * that takes the value itself (see convertingUseOf), or a destination that * holds a missing element (see assertedElementKeepsUndefined) other than a diff --git a/packages/compiler/src/frontend/lowering/lower-inspect.ts b/packages/compiler/src/frontend/lowering/lower-inspect.ts index f1ccaebc8..1ff19782c 100644 --- a/packages/compiler/src/frontend/lowering/lower-inspect.ts +++ b/packages/compiler/src/frontend/lowering/lower-inspect.ts @@ -1,5 +1,6 @@ import { keyedHelperName } from "./stable-names.js"; import { perKeyOrder } from "./key-order-witness.js"; +import { elementMayBeForeign } from "./array-values.js"; import { classPrototypeView } from "./class-prototypes.js"; import { dynUndefinedExpr, @@ -1421,11 +1422,29 @@ function inspectHelper(lowerer: Lowerer, t: IrType, loc: SrcLoc): string { const i = (): IrExpr => varRef("i.0", F64, loc); const next = (): IrExpr => varRef("next.0", F64, loc); const hasMore = (): IrExpr => lt(i(), n()); + // A foreign element (state 3) holds a value outside the element type. + const foreignElements = elementMayBeForeign(lowerer, t.elem); + const foreign = (): IrExpr => ({ + kind: "libCall", + fn: "array.foreign", + args: [v(), i()], + type: DYN, + loc, + }); const isNumber = (): IrExpr => ({ kind: "ternary", cond: eq(state(i()), numLit(1, loc)), then: isNumberFlag(lowerer, t.elem, () => at(i()), loc), - else_: boolLit(false, loc), + else_: foreignElements + ? { + kind: "ternary", + cond: eq(state(i()), numLit(3, loc)), + then: isNumberFlag(lowerer, DYN, foreign, loc), + else_: boolLit(false, loc), + type: BOOL, + loc, + } + : boolLit(false, loc), type: BOOL, loc, }); @@ -1504,7 +1523,16 @@ function inspectHelper(lowerer: Lowerer, t: IrType, loc: SrcLoc): string { kind: "ternary", cond: eq(state(i()), numLit(1, loc)), then: child(t.elem, at(i())), - else_: strLit("undefined", loc), + else_: foreignElements + ? { + kind: "ternary", + cond: eq(state(i()), numLit(3, loc)), + then: child(DYN, foreign()), + else_: strLit("undefined", loc), + type: STRING, + loc, + } + : strLit("undefined", loc), type: STRING, loc, }, diff --git a/packages/compiler/src/frontend/lowering/lower-server.ts b/packages/compiler/src/frontend/lowering/lower-server.ts index 7359b56a8..614d6a206 100644 --- a/packages/compiler/src/frontend/lowering/lower-server.ts +++ b/packages/compiler/src/frontend/lowering/lower-server.ts @@ -2425,7 +2425,9 @@ function lowerNetSocketMethodCall( }; } if (name === "write" || name === "end") { - requireStatementPosition(lowerer, call, `socket.${name}(...)`); + // write() answers Node's backpressure boolean; end() stays a statement + if (name === "end") requireStatementPosition(lowerer, call, `socket.${name}(...)`); + const resultT = name === "write" ? BOOL : VOID; // write(chunk, encoding) — the two-argument encoding form: 'buffer' // beside a string chunk is Writable's encoding check (a synchronous // ERR_UNKNOWN_ENCODING TypeError, "Unknown encoding: buffer" — the @@ -2451,7 +2453,7 @@ function lowerNetSocketMethodCall( const socket = coerceToHandle(lowerer, access.expression, NETSOCKET_T); const chunk = lowerer.lowerExpr(args[0]!); const fn: IrLibFn = chunk.type.kind === "string" ? "net.sockWrite" : "net.sockWriteBytes"; - return { kind: "libCall", fn, args: [socket, chunk], type: VOID, loc }; + return { kind: "libCall", fn, args: [socket, chunk], type: BOOL, loc }; } } } @@ -2510,7 +2512,7 @@ function lowerNetSocketMethodCall( NARROW_DATA_HINT, ); } - main = { kind: "libCall", fn, args: [receiver(), data], type: VOID, loc }; + main = { kind: "libCall", fn, args: [receiver(), data], type: resultT, loc }; } const cb = lowerer.lowerExpr(last); let register: IrExpr; @@ -2531,8 +2533,24 @@ function lowerNetSocketMethodCall( loc, }; } - stmts.push({ kind: "exprStmt", expr: main, loc }); - return { kind: "seqExpr", stmts, result: register, type: VOID, loc }; + if (resultT.kind === "void") { + stmts.push({ kind: "exprStmt", expr: main, loc }); + return { kind: "seqExpr", stmts, result: register, type: VOID, loc }; + } + // write(data, cb): the data is written, the callback registered, and + // the write's answer is the value + const okLocal = lowerer.declareHiddenLocal("%sockWriteOk", BOOL); + stmts.push( + { kind: "varDecl", localId: okLocal.id, init: main, loc }, + { kind: "exprStmt", expr: register, loc }, + ); + return { + kind: "seqExpr", + stmts, + result: { kind: "varRef", localId: okLocal.id, type: BOOL, loc }, + type: BOOL, + loc, + }; } const maxArgs = name === "write" ? 1 : 1; const minArgs = name === "write" ? 1 : 0; @@ -2552,17 +2570,17 @@ function lowerNetSocketMethodCall( const data = lowerer.lowerExpr(args[0]!); if (data.type.kind === "string") { const fn: IrLibFn = name === "write" ? "net.sockWrite" : "net.sockEndStr"; - return { kind: "libCall", fn, args: [receiver, data], type: VOID, loc }; + return { kind: "libCall", fn, args: [receiver, data], type: resultT, loc }; } if (data.type.kind === "bytes" && data.type.elem === "u8") { const fn: IrLibFn = name === "write" ? "net.sockWriteBytes" : "net.sockEndBytes"; - return { kind: "libCall", fn, args: [receiver, data], type: VOID, loc }; + return { kind: "libCall", fn, args: [receiver, data], type: resultT, loc }; } if (data.type.kind === "dyn") { // An untyped JS payload (a helper's parameter): the runtime // dispatches STR/BYTES and throws Node's chunk TypeError otherwise. const fn: IrLibFn = name === "write" ? "net.sockWriteDyn" : "net.sockEndDyn"; - return { kind: "libCall", fn, args: [receiver, data], type: VOID, loc }; + return { kind: "libCall", fn, args: [receiver, data], type: resultT, loc }; } lowerer.noLowering( `${name} of '${lowerer.fmt(data.type)}' data`, @@ -2944,6 +2962,7 @@ function lowerNetSocketMethodCall( event === "connect" || event === "ready" || event === "timeout" || + event === "drain" || event === "readable" ) { const { cb } = lowerCallbackArg( @@ -2962,11 +2981,13 @@ function lowerNetSocketMethodCall( ? "net.sockOnClose" : event === "timeout" ? "net.sockOnTimeout" - : event === "readable" - ? "net.sockOnReadable" - : event === "ready" - ? "net.sockOnReady" - : "net.sockOnConnect"; + : event === "drain" + ? "net.sockOnDrain" + : event === "readable" + ? "net.sockOnReadable" + : event === "ready" + ? "net.sockOnReady" + : "net.sockOnConnect"; return { kind: "libCall", fn, args: [receiver, cb, once], type: VOID, loc }; } if (event === "lookup") { @@ -3957,6 +3978,18 @@ export function lowerServerProperty( const receiver = coerceToHandle(lowerer, expr.expression, NETSOCKET_T); return { kind: "libCall", fn: "net.sockBytesWritten", args: [receiver], type: F64, loc }; } + if ( + recvKind === "netSocket" && + lowerer.isStdlibMember(expr) && + (expr.name.text === "writableLength" || expr.name.text === "writableNeedDrain") + ) { + // The write buffer's unsent bytes, and whether a write answered false + // and 'drain' has not fired yet. + const receiver = coerceToHandle(lowerer, expr.expression, NETSOCKET_T); + return expr.name.text === "writableLength" + ? { kind: "libCall", fn: "net.sockWritableLength", args: [receiver], type: F64, loc } + : { kind: "libCall", fn: "net.sockWritableNeedDrain", args: [receiver], type: BOOL, loc }; + } if (recvKind === "netSocket" && lowerer.isStdlibMember(expr) && expr.name.text === "readable") { // true until the read half is done (peer FIN / destroy). const receiver = coerceToHandle(lowerer, expr.expression, NETSOCKET_T); diff --git a/packages/compiler/src/frontend/lowering/lower-stmts.ts b/packages/compiler/src/frontend/lowering/lower-stmts.ts index d6d8c1cc7..9586727b3 100644 --- a/packages/compiler/src/frontend/lowering/lower-stmts.ts +++ b/packages/compiler/src/frontend/lowering/lower-stmts.ts @@ -2668,7 +2668,7 @@ export function lowerBindingPattern( lowerer.unsupported("SC1031", el, "rest elements over a union of array or tuple layouts"); const loc = locOf(el); let value = unionArrayValueRead(lowerer, srcRef(), numLit(i, loc), loc, (text) => - strCharsCall(lowerer, text, loc), + strCharsCall(text, loc), ); if (!value) lowerer.unsupported( @@ -2798,7 +2798,7 @@ export function lowerBindingPattern( } // STRING sources (`const [a, b] = str`): array destructuring walks // the STRING ITERATOR — code points, astral characters whole — - // which is Array.from(string)'s own machinery (%str.chars), so the + // which is Array.from(string)'s own machinery (str.codePoints), so the // source splits once into a hidden chars array and the pattern // lowers as an array pattern over string[]. Positions, holes, // defaults (with the bounds test), rest (the remaining code points @@ -2817,7 +2817,7 @@ export function lowerBindingPattern( out.push({ kind: "varDecl", localId: chars.id, - init: strCharsCall(lowerer, srcRef(), loc), + init: strCharsCall(srcRef(), loc), loc, }); lowerer.lowerBindingPattern( @@ -9826,7 +9826,7 @@ function lowerArrayAssignInto( let srcType = tmpRef().type; // A STRING source (`[a, b] = s`): the declaration path's rule in // assignment position — the string iterator's code-point split - // (%str.chars, astral characters whole) into a hidden chars array, + // (str.codePoints, astral characters whole) into a hidden chars array, // then the pattern proceeds as an array pattern over string[] // (defaults with the bounds test, rest packing the remaining code // points, elisions; positions past the last code point inherit the @@ -9839,7 +9839,7 @@ function lowerArrayAssignInto( out.push({ kind: "varDecl", localId: chars.id, - init: strCharsCall(lowerer, tmpRef(), loc), + init: strCharsCall(tmpRef(), loc), loc, }); tmpRef = () => ({ kind: "varRef", localId: chars.id, type: charsT, loc }); @@ -11383,16 +11383,18 @@ function lowerForOfMatchAll( /** `for (const ch of s)` over a STRING: JS's string iterator, which walks * code POINTS — each pass binds the whole character (a two-unit string * for astral chars, where charAt/slice would return one surrogate half). - * Desugars over the UTF-16 cursor machinery: a hidden unit - * index reads the character with the cpAt intrinsic and advances by ITS - * length before the body runs (so `continue` never skips the advance): + * Desugars over a byte cursor into the WTF-8 data: a hidden byte offset + * reads the character that starts there with the cpAtByte intrinsic and + * advances by ITS byte count before the body runs (so `continue` never + * skips the advance): * * { const %s = ; let %i = 0; - * while (%i < %s.length) { const ch = cpAt(%s, %i); %i += ch.length; } } + * while (%i < byteLength(%s)) { const ch = cpAtByte(%s, %i); %i += byteLength(ch); } } * - * Strings are immutable, so the length is stable and there is no + * Strings are immutable, so the byte count is stable and there is no * iterator state to unwind on throw; a lone surrogate is a one-unit - * character, so every step lands on a character start. */ + * character with its own three-byte encoding, so every step lands on a + * character start. */ /** for-of over a CLASS ITERABLE (classIteratorOf's shape) — the * iterator protocol as ordinary method calls: * @@ -12087,7 +12089,7 @@ function lowerForOfString( try { const s = lowerer.declareHiddenLocal("%strof", STRING); const i = lowerer.declareHiddenLocal("%iterof", F64); - i.mutable = true; // the unit index reassigns (hidden locals default const) + i.mutable = true; // the byte cursor reassigns (hidden locals default const) const sRef = (): IrExpr => ({ kind: "varRef", localId: s.id, type: STRING, loc }); const iRef = (): IrExpr => ({ kind: "varRef", localId: i.id, type: F64, loc }); // `for (var c of str)`: the character mechanics stay on a hidden @@ -12118,7 +12120,7 @@ function lowerForOfString( localId: ch.id, init: { kind: "strIntrinsic", - method: "cpAt", + method: "cpAtByte", receiver: sRef(), args: [iRef()], type: STRING, @@ -12145,7 +12147,7 @@ function lowerForOfString( left: iRef(), right: { kind: "strIntrinsic", - method: "length", + method: "byteLength", receiver: chRef, args: [], type: F64, @@ -12177,7 +12179,7 @@ function lowerForOfString( left: iRef(), right: { kind: "strIntrinsic", - method: "length", + method: "byteLength", receiver: sRef(), args: [], type: F64, diff --git a/packages/compiler/src/frontend/lowering/lowerer.ts b/packages/compiler/src/frontend/lowering/lowerer.ts index 3d3f1ba08..76559c5b8 100644 --- a/packages/compiler/src/frontend/lowering/lowerer.ts +++ b/packages/compiler/src/frontend/lowering/lowerer.ts @@ -2400,6 +2400,11 @@ export class Lowerer { /** for-of loops over arrays that can hold neither holes nor undefined: * each pass reads the element payload directly. */ readonly presentElementLoops = new Set(); + /** Element keys (typeKey of the element type) of native arrays that can + * hold elements outside their element type, stored through dynamic views + * (SCR_ARR_FOREIGN); "*" for every array. Empty unless the program stores + * values through a dynamic value that can view a native array. */ + foreignElementKeys: ReadonlySet = new Set(); /** The fs/promises glob() call lowerForOf is lowering as a for await * head; other glob() calls have no lowering. */ fspGlobHead: ts.Node | null = null; @@ -6486,6 +6491,20 @@ export class Lowerer { let dynamicViews: Set | null = null; // Element ABIs of the native arrays that an `any` or `unknown` value is // converted to, including arrays nested in the converted value. + // Assignments, including the logical ones (`view ??= numbers`). + const assignmentOperators = new Set([ + ts.SyntaxKind.EqualsToken, + ts.SyntaxKind.QuestionQuestionEqualsToken, + ts.SyntaxKind.BarBarEqualsToken, + ts.SyntaxKind.AmpersandAmpersandEqualsToken, + ]); + // Initializers of class properties, parameter defaults and + // destructuring defaults, which convert to the binding's type. + const initializedBinding = ( + node: ts.Node, + ): node is ts.PropertyDeclaration | ts.ParameterDeclaration | ts.BindingElement => + (ts.isPropertyDeclaration(node) || ts.isParameter(node) || ts.isBindingElement(node)) && + node.initializer !== undefined; const collectDynamicViews = (): Set => { const keys = new Set(); const dynamicValue = (node: ts.Expression): boolean => { @@ -6536,14 +6555,18 @@ export class Lowerer { dynamicValue(node.initializer) ) target(typeOrNull(node.name)); + else if (initializedBinding(node) && dynamicValue(node.initializer!)) + target(typeOrNull(node.name ?? node)); else if ( ts.isBinaryExpression(node) && - node.operatorToken.kind === ts.SyntaxKind.EqualsToken && + assignmentOperators.has(node.operatorToken.kind) && dynamicValue(node.right) ) target(typeOrNull(node.left)); else if (ts.isReturnStatement(node) && node.expression && dynamicValue(node.expression)) target(contextualOrNull(node.expression)); + else if (ts.isYieldExpression(node) && node.expression && dynamicValue(node.expression)) + target(contextualOrNull(node.expression)); else if (ts.isCallExpression(node) || ts.isNewExpression(node)) { for (const arg of node.arguments?.nodes ?? []) if (!ts.isSpreadElement(arg) && dynamicValue(arg)) target(contextualOrNull(arg)); @@ -6572,6 +6595,237 @@ export class Lowerer { }); return keys; }; + // Element ABIs of the native arrays a value converted to a dynamic type + // can reach: the conversion keeps each such array as a live view, so + // stores through the dynamic value write the native storage. + // Standard library functions that only read the arguments they take as + // `any` or `unknown`: the console methods, String, Number, Boolean, + // Array.isArray, Object.is, JSON.stringify without a replacer function, + // and util's inspect and format. Any other function can keep an + // argument (Promise.resolve, Map.prototype.set, push onto an + // `unknown[]`) or pass it to a callback (a JSON.stringify replacer, an + // emitted event), which can write through it. + const readerGlobals = new Map | null>([ + ["console", null], + ["JSON", new Set(["stringify"])], + ["Array", new Set(["isArray"])], + ["Object", new Set(["is"])], + ]); + const readerFunctions = new Set(["String", "Number", "Boolean"]); + const utilReaders = new Set(["inspect", "format"]); + const standardLibraryReader = (node: ts.CallExpression | ts.NewExpression): boolean => + panicSafe(() => { + if (!ts.isCallExpression(node)) return false; + const callee = peel(node.expression); + if (ts.isIdentifier(callee)) + return [...readerFunctions].some((name) => this.isStdlibGlobal(callee, name)); + if (!ts.isPropertyAccessExpression(callee)) return false; + const member = callee.name.text; + for (const [global, members] of readerGlobals) { + if (!this.isStdlibGlobal(callee.expression, global)) continue; + if (members !== null && !members.has(member)) return false; + if (global !== "JSON") return true; + const replacer = node.arguments.nodes[1]; + return ( + replacer === undefined || + ts.isArrayLiteralExpression(peel(replacer)) || + peel(replacer).kind === ts.SyntaxKind.NullKeyword + ); + } + if (!utilReaders.has(member)) return false; + const declaration = this.checker.getResolvedSignature(node)?.declaration; + if (declaration === undefined) return false; + const file = ts.getSourceFileOfNode(declaration); + return this.isStdlibFile(file) && /[\\/]util\.d\.ts$/.test(file.fileName); + }, false); + const collectNativeViews = (): Set => { + const keys = new Set(); + const seen = new Set(); + const collect = (type: IrType | null): void => { + if (!type) return; + const id = typeKey(type); + if (seen.has(id)) return; + seen.add(id); + if (type.kind === "array") { + if (type.elem.kind !== "dyn") keys.add(typeKey(type.elem)); + collect(type.elem); + } else if (type.kind === "union") { + for (const arm of this.unions.get(type.unionId)?.arms ?? []) collect(arm); + } else if (type.kind === "record") { + for (const field of this.shapes.get(type.shapeId)?.fields ?? []) collect(field.type); + } else if (type.kind === "object") { + for (const field of this.classes.get(type.className)?.fields.values() ?? []) + collect(field); + } else if (type.kind === "map") { + collect(type.key); + collect(type.value); + } else if (type.kind === "set") collect(type.elem); + else if (type.kind === "promise") collect(type.inner); + }; + const holdsDynamic = (type: IrType | null, depth: number): boolean => { + if (!type || depth > 4) return false; + if (type.kind === "dyn") return true; + if (type.kind === "array" || type.kind === "set") return holdsDynamic(type.elem, depth + 1); + if (type.kind === "map") return holdsDynamic(type.value, depth + 1); + if (type.kind === "promise") return holdsDynamic(type.inner, depth + 1); + if (type.kind === "union") + return (this.unions.get(type.unionId)?.arms ?? []).some((arm) => + holdsDynamic(arm, depth + 1), + ); + if (type.kind === "record") + return (this.shapes.get(type.shapeId)?.fields ?? []).some((field) => + holdsDynamic(field.type, depth + 1), + ); + return false; + }; + const mapped = (type: ts.Type | null | undefined): IrType | null => + type ? panicSafe(() => this.mapTypeOf(type), null) : null; + const flow = (source: ts.Expression, target: ts.Type | null | undefined): void => { + if (target === undefined) return; + const into = target === null ? null : mapped(target); + // An unanswerable target may be dynamic. + if (target !== null && into !== null && !holdsDynamic(into, 0)) return; + const from = typeOrNull(source); + if (from === null) keys.add("*"); + else collect(mapped(from)); + }; + const visit = (node: ts.Node): void => { + if (ts.isAsExpression(node) || ts.isTypeAssertion(node)) + flow(node.expression, typeOrNull(node)); + else if (ts.isVariableDeclaration(node) && node.type && node.initializer) + flow(node.initializer, typeOrNull(node.name)); + else if (initializedBinding(node)) flow(node.initializer!, typeOrNull(node.name ?? node)); + else if (ts.isBinaryExpression(node) && assignmentOperators.has(node.operatorToken.kind)) + flow(node.right, typeOrNull(node.left)); + else if (ts.isReturnStatement(node) && node.expression) + flow(node.expression, contextualOrNull(node.expression)); + else if (ts.isYieldExpression(node) && node.expression) + flow(node.expression, contextualOrNull(node.expression)); + // A thrown value reaches a catch binding, which is dynamic. + else if (ts.isThrowStatement(node)) flow(node.expression, null); + else if (ts.isCallExpression(node) || ts.isNewExpression(node)) { + if (!standardLibraryReader(node)) + for (const arg of node.arguments?.nodes ?? []) + flow(ts.isSpreadElement(arg) ? arg.expression : arg, contextualOrNull(arg)); + } else if (ts.isPropertyAssignment(node)) + flow(node.initializer, contextualOrNull(node.initializer)); + else if (ts.isShorthandPropertyAssignment(node)) { + const name = node.name as ts.Expression; + flow(name, contextualOrNull(name)); + } else if (ts.isArrayLiteralExpression(node)) { + for (const element of node.elements.nodes) + if (!ts.isSpreadElement(element) && !ts.isOmittedExpression(element)) + flow(element, contextualOrNull(element)); + } else if (ts.isArrowFunction(node) && !ts.isBlock(node.body)) + flow(node.body as ts.Expression, contextualOrNull(node.body as ts.Expression)); + }; + for (const sf of sourceFiles) + ts.walkPreorder(sf, (node) => { + // A checker panic hides this conversion: any array. + if (!panicSafe(() => (visit(node), true), false)) keys.add("*"); + return undefined; + }); + return keys; + }; + // A receiver typed `any`, `unknown` or an array of either can be a live + // view of a native array. A receiver of another type (`Worker[]`, + // `Record`) is one only after a lying assertion, even + // when its values are dynamic. + const dynamicType = (type: ts.Type): boolean => { + if ((type.flags & (ts.TypeFlags.Any | ts.TypeFlags.Unknown)) !== 0) return true; + if (!this.checker.isArrayType(type) || this.checker.isTupleType(type)) return false; + const element = this.checker.getTypeArguments(type as ts.TypeReference)[0]; + return ( + element !== undefined && (element.flags & (ts.TypeFlags.Any | ts.TypeFlags.Unknown)) !== 0 + ); + }; + // A binding that only ever holds the array its declaration creates (a + // const variable, or a readonly property with an initializer) is never + // a view of a native array, even when its elements are dynamic. + const freshDynamicStorage = (node: ts.Expression, depth: number): boolean => { + const e = peel(node); + if (ts.isArrayLiteralExpression(e)) return true; + if (ts.isNewExpression(e) || ts.isCallExpression(e)) { + const callee = peel(e.expression); + if (ts.isIdentifier(callee)) return this.isStdlibGlobal(callee, "Array"); + return ( + ts.isPropertyAccessExpression(callee) && + (callee.name.text === "of" || callee.name.text === "from") && + this.isStdlibGlobal(callee.expression, "Array") + ); + } + if (depth > 4 || (!ts.isIdentifier(e) && !ts.isPropertyAccessExpression(e))) return false; + // A typed array along the way is the array a view shows. + const type = typeOrNull(e); + if (type === null || !dynamicType(type)) return false; + const symbol = panicSafe( + () => this.checker.getSymbolAtLocation(ts.isIdentifier(e) ? e : e.name), + undefined, + ); + const declaration = symbol?.valueDeclaration; + if (!symbol || !declaration) return false; + // Variables and class properties only: parameters take any argument, + // and object literals fill interface properties. + if ( + !(ts.isVariableDeclaration(declaration) && ts.isIdentifier(declaration.name)) && + !ts.isPropertyDeclaration(declaration) + ) + return false; + const assigned = assignedValues(symbol); + if (assigned === null) return false; + const values = [ + ...(declaration.initializer ? [declaration.initializer] : []), + ...(assigned ?? []), + ]; + return values.length > 0 && values.every((value) => freshDynamicStorage(value, depth + 1)); + }; + // Every value `=` assigns to each variable or property of the program, + // by symbol; null for one also written another way (compound or + // destructuring assignment). Built on first use. + let assignments: Map | null = null; + const assignedValues = (symbol: ts.Symbol): ts.Expression[] | null | undefined => { + if (assignments === null) { + const found = new Map(); + const add = (target: ts.Expression, value: ts.Expression | null): void => { + const e = peel(target); + if (ts.isArrayLiteralExpression(e) || ts.isObjectLiteralExpression(e)) { + // Destructuring: every target is written with a part of the value. + ts.forEachChild(e, (child) => { + if (ts.isIdentifier(child) || ts.isPropertyAccessExpression(child)) add(child, null); + else if (ts.isSpreadElement(child)) add(child.expression, null); + else if (ts.isPropertyAssignment(child)) add(child.initializer, null); + else if (ts.isShorthandPropertyAssignment(child)) + add(child.name as ts.Expression, null); + else if (ts.isBinaryExpression(child)) add(child.left, null); + return undefined; + }); + return; + } + const name = ts.isIdentifier(e) ? e : ts.isPropertyAccessExpression(e) ? e.name : null; + const written = name + ? panicSafe(() => this.checker.getSymbolAtLocation(name), undefined) + : undefined; + if (!written) return; + const list = found.get(written); + if (list === null) return; + if (value === null) found.set(written, null); + else if (list) list.push(value); + else found.set(written, [value]); + }; + for (const sf of sourceFiles) + ts.walkPreorder(sf, (node) => { + if (ts.isBinaryExpression(node)) { + const op = node.operatorToken.kind; + if (op === ts.SyntaxKind.EqualsToken) add(node.left, node.right); + else if (op >= ts.SyntaxKind.FirstAssignment && op <= ts.SyntaxKind.LastAssignment) + add(node.left, null); + } + return undefined; + }); + assignments = found; + } + return assignments.get(symbol); + }; // `t[i]!` on a typed array, stored into an array, keeps an invalid // index's undefined (assertedReadObservesUndefined): `!` performs no // runtime check. A read whose consumers treat undefined like NaN stores @@ -6589,6 +6843,11 @@ export class Lowerer { ); }; const elementStates = new ArrayElementStates({ + isDynamicArray: (node) => { + const type = typeOrNull(node); + return (type === null || dynamicType(type)) && !freshDynamicStorage(node, 0); + }, + nativeViewKeys: collectNativeViews, elementAdmitsUndefined: (key) => { siteDependencies?.read(key); return optionalElementKeys.has(key); @@ -7509,6 +7768,7 @@ export class Lowerer { } for (const [site, present] of presentSites) if (present) this.presentElementCalls.add(site); for (const [loop, present] of presentLoops) if (present) this.presentElementLoops.add(loop); + this.foreignElementKeys = elementStates.foreignKeys(); const globalsById = new Map(this.globalsList.map((global) => [global.id, global])); for (const [symbol, field] of staticFieldsBySymbol) { if (!optionalSymbols.has(symbol)) continue; @@ -8062,6 +8322,9 @@ export class Lowerer { ...(this.usesLifetimeRegions ? { lifetimeRegions: true as const } : {}), ...(this.usesNodeIsland ? { nodeIsland: this.nodeIslandRecord() } : {}), ...(this.declBoundary ? { declBoundary: true as const } : {}), + ...(this.foreignElementKeys.size > 0 + ? { foreignElementKeys: [...this.foreignElementKeys].sort() } + : {}), ...(this.threadSlotKeys.size > 0 ? { threadSlotKeys: [...this.threadSlotKeys].sort() } : {}), @@ -10644,8 +10907,8 @@ export class Lowerer { return unionRetagHelper(this, fromId, toId, loc, trappable); } - narrowedArmHelper(fromId: string, target: IrType, loc: SrcLoc): string | null { - return narrowedArmHelper(this, fromId, target, loc); + narrowedArmHelper(fromId: string, target: IrType, loc: SrcLoc, message?: string): string | null { + return narrowedArmHelper(this, fromId, target, loc, message); } checkedNarrowCall(helper: string, value: IrExpr, type: IrType, loc: SrcLoc): IrExpr { diff --git a/packages/compiler/src/frontend/lowering/runtime-optional-elements.ts b/packages/compiler/src/frontend/lowering/runtime-optional-elements.ts index 7561f4a65..0840cef91 100644 --- a/packages/compiler/src/frontend/lowering/runtime-optional-elements.ts +++ b/packages/compiler/src/frontend/lowering/runtime-optional-elements.ts @@ -43,8 +43,17 @@ export interface ElementStateHost { ownerOf(node: ts.Expression): ts.Symbol | null; /** The owning slot a newly built array is stored into. */ allocationOwner(node: ts.Expression): ts.Symbol | null; + /** `node` is a dynamic value (`any`, `unknown`, `unknown[]`): when it is + * a live view of a native array, a store through it writes that array's + * storage. */ + isDynamicArray(node: ts.Expression): boolean; + /** Element keys of the native arrays a value converted to a dynamic type + * can reach, as live views ("*" for any array). */ + nativeViewKeys(): ReadonlySet; /** Debug provenance for a new fact. */ - note: ((fact: "undefined" | "holes", key: string, site: ts.Node | undefined) => void) | null; + note: + | ((fact: "undefined" | "holes" | "foreign", key: string, site: ts.Node | undefined) => void) + | null; /** The scan read, or added to, the facts of an element key, of an owning * slot, or of every array (ANY_ARRAY), for dependency tracking * (SiteDependencies). */ @@ -54,6 +63,16 @@ export interface ElementStateHost { /** The fact name of the facts shared by every array. */ const ANY_ARRAY = "*"; +/** The fact name of holes stored through dynamic views. */ +const VIEW_HOLES = "%view"; +/** The fact name of values stored through dynamic views. */ +const VIEW_VALUES = "%view-values"; + +/** Element keys whose arrays store a value written through a dynamic view + * that the element type cannot hold, as a foreign element: `number[]` and + * `string[]`. Arrays of other element types reject such a write with the + * checked conversion's TypeError. */ +const FOREIGN_ELEMENT_KEYS: ReadonlySet = new Set(["f64", "string"]); /** What an array value can contain besides present values. */ interface Contents { @@ -87,6 +106,13 @@ export class ArrayElementStates { * whose element ABI is decided per generic instantiation. */ private anyUndefined = false; private anyHoley = false; + /** A store through a dynamic value can leave holes; they land in every + * native array such a value views. */ + private viewHoles = false; + /** A store through a dynamic value can write any value, which the + * viewed native arrays then hold as undefined or foreign elements. */ + private viewValues = false; + private viewKeys: ReadonlySet | null = null; /** Facts of arrays owned by one storage slot. */ private readonly undefinedOwners = new Set(); private readonly holeyOwners = new Set(); @@ -336,14 +362,65 @@ export class ArrayElementStates { this.host.read?.(ANY_ARRAY); if (keys === null) return null; for (const key of keys) this.host.read?.(key); + this.host.read?.(VIEW_HOLES); + this.host.read?.(VIEW_VALUES); return { undefined: this.anyUndefined || - keys.some((key) => this.undefinedKeys.has(key) || this.host.elementAdmitsUndefined(key)), - holes: this.anyHoley || keys.some((key) => this.holeyKeys.has(key)), + keys.some( + (key) => + this.undefinedKeys.has(key) || + this.host.elementAdmitsUndefined(key) || + this.viewedValues(key), + ), + holes: this.anyHoley || keys.some((key) => this.holeyKeys.has(key) || this.viewed(key)), }; } + /** Holes stored through a dynamic view can be in arrays of this key. */ + private viewed(key: string): boolean { + return this.viewHoles && this.isViewed(key); + } + + /** Values of any type stored through a dynamic view can be in arrays of + * this key. */ + private viewedValues(key: string): boolean { + return this.viewValues && FOREIGN_ELEMENT_KEYS.has(key) && this.isViewed(key); + } + + private isViewed(key: string): boolean { + this.viewKeys ??= this.host.nativeViewKeys(); + return this.viewKeys.has(key) || this.viewKeys.has(ANY_ARRAY); + } + + /** Element keys whose arrays can hold elements outside their element + * type, stored through dynamic views (of FOREIGN_ELEMENT_KEYS). Final + * once the fixed point is reached. */ + foreignKeys(): ReadonlySet { + if (!this.viewValues) return new Set(); + return new Set([...FOREIGN_ELEMENT_KEYS].filter((key) => this.isViewed(key))); + } + + /** A store through `receiver`, which has no native element key, can + * write a value of any type into a viewed native array. */ + private addViewValues(receiver: ts.Expression): boolean { + if (this.viewValues || !this.host.isDynamicArray(receiver)) return false; + this.viewValues = true; + this.host.changed?.(VIEW_VALUES); + this.host.note?.("foreign", VIEW_VALUES, this.site); + return true; + } + + /** A store through `receiver`, which has no native element key, can + * leave holes. Only a dynamic value can be a view of a native array. */ + private addViewHoles(receiver: ts.Expression): boolean { + if (this.viewHoles || !this.host.isDynamicArray(receiver)) return false; + this.viewHoles = true; + this.host.changed?.(VIEW_HOLES); + this.host.note?.("holes", VIEW_HOLES, this.site); + return true; + } + /** Reading every index of `node` (a spread, an iteration, `Array.from`) * can produce `undefined`. Only native arrays can: other iterables reject * an undefined value when it is stored. */ @@ -437,7 +514,11 @@ export class ArrayElementStates { /** A store through `receiver` can leave holes. */ private addHolesTo(receiver: ts.Expression): boolean { const owner = this.host.ownerOf(receiver); - if (owner === null) return this.addHoles(this.host.arrayKeys(receiver)); + if (owner === null) { + const keys = this.host.arrayKeys(receiver); + if (keys !== null && keys.length === 0) return this.addViewHoles(receiver); + return this.addHoles(keys); + } if (this.holeyOwners.has(owner)) return false; this.holeyOwners.add(owner); this.host.changed?.(owner); @@ -497,10 +578,18 @@ export class ArrayElementStates { op === ts.SyntaxKind.QuestionQuestionEqualsToken || op === ts.SyntaxKind.BarBarEqualsToken || op === ts.SyntaxKind.AmpersandAmpersandEqualsToken; + const keys = this.host.arrayKeys(target.expression); + if (keys !== null && keys.length === 0) { + // Through a dynamic value: any element write can store a value the + // viewed native array's element type does not hold. + if (namesProperty(target)) return false; + let changed = false; + if (holes && this.addViewHoles(target.expression)) changed = true; + if (this.addViewValues(target.expression)) changed = true; + return changed; + } const undefinedValue = storesValue && this.host.mayBeUndefined(node.right); if (!holes && !undefinedValue) return false; - const keys = this.host.arrayKeys(target.expression); - if (keys !== null && keys.length === 0) return false; let changed = false; if (holes && this.addHolesTo(target.expression)) changed = true; if (undefinedValue && this.addUndefinedTo(target.expression)) changed = true; @@ -511,8 +600,6 @@ export class ArrayElementStates { target.name.text === "length" && !isShrinkingLength(node, target) ) { - const keys = this.host.arrayKeys(target.expression); - if (keys !== null && keys.length === 0) return false; return this.addHolesTo(target.expression); } return false; @@ -563,7 +650,14 @@ export class ArrayElementStates { return changed; } const receiver = this.host.arrayKeys(callee.expression); - if (receiver !== null && receiver.length === 0) return changed; + if (receiver !== null && receiver.length === 0) { + if ( + (stored.length > 0 || callee.name.text === "fill") && + this.addViewValues(callee.expression) + ) + changed = true; + return changed; + } for (const value of stored) if (this.valueMayBeUndefined(value) && this.addUndefinedTo(callee.expression)) changed = true; return changed; @@ -596,10 +690,17 @@ export class ArrayElementStates { )); if (!reflective) return false; let changed = false; - for (const arg of args) { + // The written object: `Reflect.apply`'s receiver argument, otherwise + // the first argument. + const written = this.host.isGlobal(owner, "Reflect") && method === "apply" ? 1 : 0; + for (const [index, arg] of args.entries()) { const target = ts.isSpreadElement(arg) ? arg.expression : arg; const keys = this.host.arrayKeys(target); - if (keys !== null && keys.length === 0) continue; + if (keys !== null && keys.length === 0) { + if (index === written && this.addViewHoles(target)) changed = true; + if (index === written && this.addViewValues(target)) changed = true; + continue; + } if (this.addUndefinedTo(target)) changed = true; if (this.addHolesTo(target)) changed = true; } @@ -783,6 +884,14 @@ function isAppendIndex(target: ts.ElementAccessExpression): boolean { ); } +/** `o["name"]`: a string literal key that is not an array index writes a + * named property, never an element. */ +function namesProperty(target: ts.ElementAccessExpression): boolean { + const key = peel(target.argumentExpression); + if (!ts.isStringLiteral(key) && !ts.isNoSubstitutionTemplateLiteral(key)) return false; + return !/^(0|[1-9][0-9]*)$/.test(key.text) || Number(key.text) >= 2 ** 32 - 1; +} + /** `xs.length = 0`, `xs.length = xs.length - k` and `xs.length -= k` * truncate; truncation never leaves holes. */ function isShrinkingLength( diff --git a/packages/compiler/src/frontend/lowering/union-narrow.test.ts b/packages/compiler/src/frontend/lowering/union-narrow.test.ts index ed60fac8d..5c2a81b5e 100644 --- a/packages/compiler/src/frontend/lowering/union-narrow.test.ts +++ b/packages/compiler/src/frontend/lowering/union-narrow.test.ts @@ -97,3 +97,22 @@ test("a deferred read requires a real undefined tag", () => { const value: IrExpr = { kind: "boolLit", value: false, type: BOOL, loc }; expect(buildUnionNarrow("bad", from, BOOL, loc, () => "", value)).toBeNull(); }); + +test("a failure message replaces every other tag's TypeError", () => { + const fn = buildUnionNarrow( + "call", + union, + STRING, + loc, + (type) => type.kind, + null, + "f is not a function", + )!; + const thrown = branches(fn).flatMap((branch) => { + const statement = branch.then[0]!; + return statement.kind === "throw" ? [statement.value] : []; + }); + expect(thrown).toHaveLength(4); + for (const value of thrown) + expect(value).toMatchObject({ kind: "libCall", args: [{ value: "f is not a function" }] }); +}); diff --git a/packages/compiler/src/frontend/lowering/union-narrow.ts b/packages/compiler/src/frontend/lowering/union-narrow.ts index 3bb09cb26..a167631fc 100644 --- a/packages/compiler/src/frontend/lowering/union-narrow.ts +++ b/packages/compiler/src/frontend/lowering/union-narrow.ts @@ -20,7 +20,10 @@ export function unionNarrowFailureMessage( * * Deferred scalar fields can supply the established undefined default: * false for booleans or NaN for numbers. No other tag receives that default, - * and reference-bearing targets always use checked extraction. */ + * and reference-bearing targets always use checked extraction. + * + * `failureMessage` replaces the TypeError message of every other tag (the + * error Node raises for the operation consuming the value). */ export function buildUnionNarrow( name: string, from: IrUnionDef, @@ -28,6 +31,7 @@ export function buildUnionNarrow( loc: SrcLoc, formatType: (type: IrType) => string, undefinedDefault: IrExpr | null = null, + failureMessage: string | null = null, ): IrFunction | null { if (isUnitType(target) || target.kind === "void") return null; const targetTag = from.arms.findIndex((arm) => typeEquals(arm, target)); @@ -66,7 +70,7 @@ export function buildUnionNarrow( args: [ { kind: "strLit", - value: unionNarrowFailureMessage(arm, formatType), + value: failureMessage ?? unionNarrowFailureMessage(arm, formatType), type: STRING, loc, }, diff --git a/packages/compiler/src/frontend/npm-static-type-exports-host.ts b/packages/compiler/src/frontend/npm-static-type-exports-host.ts index 8a61ed62b..f8aaa20f9 100644 --- a/packages/compiler/src/frontend/npm-static-type-exports-host.ts +++ b/packages/compiler/src/frontend/npm-static-type-exports-host.ts @@ -2,7 +2,8 @@ * package's declaration entry and runtime entry, copies the declaration * files the entry reaches, and builds the runtime entry's typedefs. */ import { dirname, relative, resolve as resolvePath } from "node:path"; -import { trackedFileExists, trackedReadFile } from "./input-tracker.js"; +import { trackedFileExists } from "./input-tracker.js"; +import { packageSourceText } from "./type-patches/layer.js"; import { resolveBareModule } from "./resolve.js"; import { isJsSourceFileName } from "./tsc-codes.js"; import { NPM_STATIC_TYPES_SUFFIX } from "./npm-static.js"; @@ -78,7 +79,7 @@ export function npmStaticTypeExportPlan( const parse = (file: string): ts.SourceFile | null => { let sf = parsed.get(file); if (sf === undefined) { - const text = trackedReadFile(file); + const text = packageSourceText(file); sf = text === null ? null : services.parse(file, text, "ts"); parsed.set(file, sf); } @@ -212,7 +213,7 @@ export function npmStaticTypeExportPlan( continue; const runtime = resolveBareModule(entryPath, pkg, "js-only"); if (runtime === null || !isJsSourceFileName(runtime.typesFile)) continue; - const runtimeText = trackedReadFile(runtime.typesFile); + const runtimeText = packageSourceText(runtime.typesFile); if (runtimeText === null) continue; const { types, values } = typesOf(declared.typesFile, new Set()); const declaredByRuntime = runtimeTypeNames(services, runtime.typesFile, runtimeText); diff --git a/packages/compiler/src/frontend/npm-static.ts b/packages/compiler/src/frontend/npm-static.ts index f51c6bbe2..ed442be79 100644 --- a/packages/compiler/src/frontend/npm-static.ts +++ b/packages/compiler/src/frontend/npm-static.ts @@ -23,6 +23,7 @@ import { trackedRealpath, } from "./input-tracker.js"; import { npmBuiltinAliasOf } from "./builtin-modules.js"; +import { packageSourceText } from "./type-patches/layer.js"; let activePackages: ReadonlySet = new Set(); let declarationOverloads: ReadonlyMap = new Map(); @@ -82,7 +83,7 @@ export function setNpmStaticTypeExports( function withTypeExportTypedefs(path: string, text: string | null): string | null { const typedefs = typeExportTypedefs.get(path.split("\\").join("/")); if (typedefs === undefined) return text; - const source = text ?? trackedReadFile(path); + const source = text ?? packageSourceText(path); return source === null ? text : `${source}\n${typedefs}\n`; } @@ -467,7 +468,11 @@ export function npmStaticFsShadow(services: FrontendServices): NpmStaticFsShadow if (hit !== undefined) return hit ?? undefined; let rewritten: string | null = null; try { - let original = trackedReadFile(path); + // A project type patch (type-patches/layer.ts) edits the text + // as the package wrote it, before any rewrite reads its JSDoc + // (unrewritten files fall through to the host, which serves the + // patched text too). + let original = packageSourceText(path); // A bundle's CommonJS chunks move to their own modules first, so // every later pass (and this shadow, for each chunk) sees plain // CommonJS module text. diff --git a/packages/compiler/src/frontend/pipeline.ts b/packages/compiler/src/frontend/pipeline.ts index aff332416..75faaa781 100644 --- a/packages/compiler/src/frontend/pipeline.ts +++ b/packages/compiler/src/frontend/pipeline.ts @@ -48,6 +48,7 @@ import { NODE_API_MODULE } from "./node-api-import.js"; import { erasedTypeOnlyImport } from "./type-only-imports.js"; import { EXPERIMENTAL_MODULE } from "./experimental-import.js"; import { readProjectConfig } from "../project-config.js"; +import { typePatchDiagnostics } from "./type-patches/layer.js"; import { packageDirOf } from "../node-islands/origin.js"; import { scanFiles, @@ -741,10 +742,14 @@ function loadFrontend( const projectConfig = readProjectConfig(entryPath); const configDiagnostics = projectConfig.diagnostics; const report = options.nodeIslands === true; + // Type patches that did not apply as written (SC8030–SC8033) warn; the + // build goes on with the packages' own declarations there. + const patchDiagnostics = typePatchDiagnostics()?.diagnostics ?? []; + const buildWarnings = [...(report ? [] : configDiagnostics), ...patchDiagnostics]; return { preflight: report && configDiagnostics.length > 0 ? [...configDiagnostics, ...preflight] : preflight, - ...(!report && configDiagnostics.length > 0 ? { warnings: configDiagnostics } : {}), + ...(buildWarnings.length > 0 ? { warnings: buildWarnings } : {}), entryText: () => finalLoad.entry.text, entryExports: () => entryFunctionExports(finalLoad.entry), // The contract scans the PROGRAM's source files, not the runtime @@ -779,9 +784,10 @@ function loadFrontend( ...finalLoad.program.getImplementationSourceFiles().filter((sf) => !sf.isDeclarationFile), ].map((sf) => [sf.fileName, sf.text]), ); - // SC8004 diagnostics point into the project's package.json. + // SC8004 and type patch diagnostics point into the project's + // package.json. if ( - configDiagnostics.length > 0 && + (configDiagnostics.length > 0 || patchDiagnostics.length > 0) && projectConfig.manifest !== null && projectConfig.manifestText !== null ) diff --git a/packages/compiler/src/frontend/program.ts b/packages/compiler/src/frontend/program.ts index 00afcd696..b93e3f217 100644 --- a/packages/compiler/src/frontend/program.ts +++ b/packages/compiler/src/frontend/program.ts @@ -140,6 +140,7 @@ import { } from "./type-only-imports.js"; import { moduleSourceCandidates } from "./module-source-candidates.js"; import { planDeclProjection, type DeclProjection } from "./decl-projection.js"; +import { configureTypePatches, packageSourceText } from "./type-patches/layer.js"; import { reportShapes } from "./shape-inference/index.js"; import { inferredJsDiagnosticSuppressed } from "./inferred-js-diagnostics.js"; import { @@ -477,7 +478,7 @@ function visitNpmStaticDeclarationClosure( if (normalized !== packageRoot && !normalized.startsWith(packageRoot + "/")) return; if (seen.has(path)) return; seen.add(path); - const source = trackedReadFile(path); + const source = packageSourceText(path); if (source === null) return; onSource(path, source); for (const specifier of npmStaticDeclarationReexports(services.parse(path, source, "ts"))) { @@ -1068,6 +1069,9 @@ export function loadProgram( // JavaScript, even when named. A registry entry's `sourceVersions` // (chalk 6) compile from their own source. configurePackageShims(entryPath); + // The project's type patches (type-patches/layer.ts), read by every host + // this load creates; nothing at all without `scriptc.typePatches`. + configureTypePatches(entryPath, (path, text, kind) => services.parse(path, text, kind)); const npmStaticPackages = [...new Set(opts?.npmStatic ?? [])].filter( (name) => !npmStaticOmitsPackage(entryPath, name), ); diff --git a/packages/compiler/src/frontend/ts7/program-host.ts b/packages/compiler/src/frontend/ts7/program-host.ts index 84cbe2474..9e8f51bb3 100644 --- a/packages/compiler/src/frontend/ts7/program-host.ts +++ b/packages/compiler/src/frontend/ts7/program-host.ts @@ -46,6 +46,7 @@ import { trackedRealpath, } from "../input-tracker.js"; import { packageShimDeclarationText } from "../package-shims.js"; +import { typePatchedText } from "../type-patches/layer.js"; import { ambientDtsPath, fallbackDtsPath, overridesDtsPath } from "../dts-paths.js"; import { reusedLibDiagnostics } from "./lib-diagnostics.js"; import { ts7Diagnostic } from "./diagnostic.js"; @@ -160,7 +161,7 @@ export class Ts7Host { const virtualFiles = this.virtualFiles; const shadow = options?.fsShadow ?? null; // A virtual file, a shadow's replacement, a package shim's declaration - // entry or the file on disk; null when shadowed out of existence or + // entry, a type-patched package file or the file on disk; null when shadowed out of existence or // missing. this.readSource = (fileName) => { const virtual = virtualFiles.get(tsgoPath(fileName)); @@ -175,6 +176,9 @@ export class Ts7Host { // program the host creates. const shim = packageShimDeclarationText(fileName); if (shim !== undefined) return shim; + // A package file a project type patch edits (type-patches/layer.ts). + const patched = typePatchedText(fileName); + if (patched !== undefined) return stripSourceBom(patched); const source = trackedReadFile(fileName); return source === null ? null : stripSourceBom(source); }; diff --git a/packages/compiler/src/frontend/type-mapper.ts b/packages/compiler/src/frontend/type-mapper.ts index 17f9d0d11..edb3cf144 100644 --- a/packages/compiler/src/frontend/type-mapper.ts +++ b/packages/compiler/src/frontend/type-mapper.ts @@ -4948,6 +4948,19 @@ function mapTypeInner(type: ts.Type, ctx: TypeMapperCtx): IrType | null { const record = mapRecordType(widened, ctx); if (record) return record; } + // An intersection whose constituents all map to ONE representation is + // that representation. An intersection that restates an optional member + // as required (`Opts & { files: Array<{ path: string }> }` over + // `files?: Array<{ path: string }>`) gives the member the type `A[] & B[]` + // with two separately written but structurally equal element literals; + // the checker keeps both, while their arrays (tuples, maps, ...) intern + // to one IR type. Reached only after every other intersection rule + // declined, so intersections that already mapped keep their mapping. A + // dynamic result stays unmapped: two opaque types are not one type. + if (widened.isIntersectionType()) { + const same = mapEqualConstituentsIntersection(widened, ctx); + if (same) return same; + } // Unions. `boolean` is internally `true | false`, and narrowing produces // literal unions like `"a" | "b"` — parts that map to the SAME IR type // collapse (deduplicated by typeKey), so a single surviving arm is just @@ -6813,6 +6826,19 @@ export function withUnitArm( * assigned keys, which is also Node's answer for Object.assign(fn, obj) * (a function's own length/name are non-enumerable). Null whenever the * shape doesn't fit — the caller falls through to the plain-func mapping. */ +/** The IR type every constituent of an intersection maps to, or null when a + * constituent is unmappable or dynamic, or two constituents differ. */ +function mapEqualConstituentsIntersection(widened: ts.Type, ctx: TypeMapperCtx): IrType | null { + let same: IrType | null = null; + for (const part of ts.constituentTypes(widened)) { + const mapped = mapType(part, ctx); + if (mapped === null || mapped.kind === "dyn") return null; + if (same === null) same = mapped; + else if (!typeEquals(same, mapped)) return null; + } + return same; +} + function mapHybridCallableIntersection(widened: ts.Type, ctx: TypeMapperCtx): IrType | null { const { checker, shapes } = ctx; if (!widened.isIntersectionType()) return null; diff --git a/packages/compiler/src/frontend/type-patches/layer.ts b/packages/compiler/src/frontend/type-patches/layer.ts new file mode 100644 index 000000000..79b409110 --- /dev/null +++ b/packages/compiler/src/frontend/type-patches/layer.ts @@ -0,0 +1,379 @@ +/* Type patches: project-owned fixes for wrong npm declarations. + * + * `scriptc.typePatches` in the entry's package.json maps + * `"@"` to declaration patches (project-config.ts): + * + * "commander@^15.0.0": { "declarations": { "HelpContext.helpWidth": "| undefined" } } + * + * The patches are applied to file TEXT, in one place every consumer reads + * through: the program host's source reads (ts7/program-host.ts), which + * serve both checker worlds (the lowering world that types package + * JavaScript from its JSDoc, and the declaration world that types user + * code at the declaration boundary), and the npm-static passes that read + * package files themselves (`packageSourceText`). Downstream, the patched + * declaration simply IS the declaration: scriptc enforces it exactly as it + * enforced the original, so a patch can never make a check weaker than the + * declaration it replaces. + * + * A patch applies to an installed copy whose version satisfies its range; + * an `@types/` twin is matched by the runtime package's name and + * installed version. Without patches nothing here runs: `typePatchedText` + * answers undefined from one null test, so every program reads the same + * bytes as before (the zero-cost rule). + * + * See scriptc-perf/results/design-type-patches/DESIGN.md. */ + +import { dirname, relative } from "node:path"; +import type { ScrDiagnostic } from "../../diagnostics/diagnostic.js"; +import { typePatchDiag } from "../../diagnostics/diagnostic.js"; +import { readProjectConfig, type TypePatchEntry } from "../../project-config.js"; +import { trackedReadFile } from "../input-tracker.js"; +import type { SourceFile } from "../ts7/syntax.js"; +import { applyPatchEdits, patchFile, type PatchMatch, type PatchRequest } from "./locate.js"; +import { parseSelector, parseTypeText, type Selector, type TypePatchOp } from "./selector.js"; +import { satisfies } from "./semver.js"; + +export type TypePatchParse = (path: string, text: string, kind: "ts" | "js") => SourceFile; + +interface Request extends PatchRequest { + entry: TypePatchEntry; + /** Offset of the selector in the manifest. */ + offset: number; +} + +interface CopyInfo { + /** The runtime package's name (for an `@types` twin, the package it + * types). */ + pkg: string; + version: string | null; + root: string; + /** Requests whose entry's range the version satisfies. */ + requests: Request[]; +} + +interface MatchRecord extends PatchMatch { + file: string; + pkg: string; + version: string | null; +} + +interface State { + manifest: string; + /** The directory of the project's package.json. */ + projectDir: string; + manifestText: string; + parse: TypePatchParse; + byPackage: Map; + copies: Map; + files: Map; + /** Package copies whose version no patch range covers. */ + unmatchedCopies: Map; + /** Requests applicable to a copy this load read files of. */ + touched: Set; + matches: MatchRecord[]; + problems: Map; + /** Patched spans per served file (patched-text coordinates). */ + spans: Map; +} + +let state: State | null = null; + +/** Registers the project's type patches for one program load (or clears + * them). Called once per load, like configurePackageShims. */ +export function configureTypePatches(entryPath: string, parse: TypePatchParse): void { + const config = readProjectConfig(entryPath); + if (config.typePatches.length === 0 || config.manifest === null) { + state = null; + return; + } + const byPackage = new Map(); + let id = 0; + for (const entry of config.typePatches) { + for (const decl of entry.declarations) { + const selector = parseSelector(decl.selector); + const op = parseTypeText(decl.type); + if ("error" in selector || "error" in op) continue; // reported as SC8004 + const list = byPackage.get(entry.pkg) ?? []; + list.push({ + id: id++, + selector: selector as Selector, + op: op as TypePatchOp, + entry, + offset: decl.offset, + }); + byPackage.set(entry.pkg, list); + } + } + state = { + manifest: config.manifest, + projectDir: dirname(config.manifest), + manifestText: config.manifestText ?? "", + parse, + byPackage, + copies: new Map(), + files: new Map(), + unmatchedCopies: new Map(), + touched: new Set(), + matches: [], + problems: new Map(), + spans: new Map(), + }; +} + +/** True while the current load has type patches. */ +export function typePatchesActive(): boolean { + return state !== null; +} + +const PATCHABLE = /\.(?:[cm]?js|d\.[cm]?ts|[cm]?ts)$/; + +/** `@types/foo__bar` types `@foo/bar`. */ +function unmangleTypesName(name: string): string { + const bare = name.slice("@types/".length); + const i = bare.indexOf("__"); + return i > 0 ? `@${bare.slice(0, i)}/${bare.slice(i + 2)}` : bare; +} + +function versionAt(dir: string): string | null { + const text = trackedReadFile(`${dir}/package.json`); + if (text === null) return null; + try { + const parsed = JSON.parse(text) as { version?: unknown }; + return typeof parsed.version === "string" ? parsed.version : null; + } catch { + return null; + } +} + +/** The version of the runtime package `pkg` an `@types` twin under + * `nodeModules` describes: the copy installed beside it, else the one the + * project resolves (pnpm keeps each package in its own directory, so a + * twin has no runtime package beside it), else null. */ +function twinRuntimeVersion(s: State, nodeModules: string, pkg: string): string | null { + const beside = versionAt(nodeModules + pkg); + if (beside !== null) return beside; + for (let dir = s.projectDir; ;) { + const version = versionAt(`${dir}/node_modules/${pkg}`); + if (version !== null) return version; + const parent = dirname(dir); + if (parent === dir) return null; + dir = parent; + } +} + +/** The installed package copy a file belongs to, with the requests that + * apply to it; null for files outside node_modules or of unpatched + * packages. */ +function copyOf(s: State, path: string): { copy: CopyInfo; rel: string } | null { + const marker = "/node_modules/"; + const at = path.lastIndexOf(marker); + if (at < 0) return null; + const after = path.slice(at + marker.length); + const parts = after.split("/"); + const nameParts = parts[0]!.startsWith("@") ? 2 : 1; + if (parts.length <= nameParts) return null; + const name = parts.slice(0, nameParts).join("/"); + const root = path.slice(0, at + marker.length) + name; + const rel = parts.slice(nameParts).join("/"); + let copy = s.copies.get(root); + if (copy === undefined) { + copy = null; + const pkg = name.startsWith("@types/") ? unmangleTypesName(name) : name; + const requests = s.byPackage.get(pkg); + if (requests !== undefined) { + // A twin is matched by the runtime package's version, never its own. + const version = name.startsWith("@types/") + ? twinRuntimeVersion(s, path.slice(0, at + marker.length), pkg) + : versionAt(root); + const applicable = requests.filter( + (r) => version !== null && satisfies(version, r.entry.range), + ); + // Later entries win on the same selector. + const bySelector = new Map(); + for (const r of applicable) bySelector.set(r.selector.text, r); + const chosen = [...bySelector.values()]; + if (chosen.length === 0) { + const entries = [...new Set(requests.map((r) => r.entry))]; + s.unmatchedCopies.set(root, { pkg, version, entries }); + } + copy = chosen.length === 0 ? null : { pkg, version, root, requests: chosen }; + } + s.copies.set(root, copy); + } + if (copy === null) return null; + for (const r of copy.requests) s.touched.add(r); + return { copy, rel }; +} + +/** Names a request's root can appear under in a file's text. */ +function rootWords(sel: Selector): string[] { + if (sel.root === "export=") return ["exports", "export"]; + if (sel.root === "default") return ["default"]; + return [sel.root]; +} + +/** The patched text of a package file, or undefined when no patch applies + * to it (callers then read the file as usual). */ +export function typePatchedText(fileName: string): string | undefined { + const s = state; + if (s === null) return undefined; + const path = fileName.split("\\").join("/"); + if (!PATCHABLE.test(path)) return undefined; + const cached = s.files.get(path); + if (cached !== undefined) return cached ?? undefined; + let result: string | null = null; + const located = copyOf(s, path); + if (located !== null) { + const text = trackedReadFile(path); + const requests = located.copy.requests.filter( + (r) => + (r.selector.file === null || r.selector.file === located.rel) && + rootWords(r.selector).some((w) => text?.includes(w)), + ); + if (text !== null && requests.length > 0) { + const kind = /\.[cm]?ts$/.test(path) ? "ts" : "js"; + let out: ReturnType | null = null; + try { + out = patchFile(located.rel, text, s.parse(path, text, kind), requests); + } catch { + out = null; // unparseable: served unpatched, like other npm-static passes + } + if (out !== null) { + const byId = new Map(requests.map((r) => [r.id, r])); + for (const m of out.matches) { + s.matches.push({ + ...m, + file: path, + pkg: located.copy.pkg, + version: located.copy.version, + }); + } + for (const p of out.problems) { + const r = byId.get(p.id); + const prior = r === undefined ? undefined : s.problems.get(r); + if (r !== undefined && (prior === undefined || (p.always === true && !prior.always))) + s.problems.set(r, { + message: `${p.message} (${located.rel})`, + always: p.always === true, + }); + } + if (out.edits.length > 0) { + result = applyPatchEdits(text, out.edits); + s.spans.set(path, patchedSpans(out.edits)); + } + } + } + } + s.files.set(path, result); + return result ?? undefined; +} + +/** Edit spans in the patched text's coordinates. */ +function patchedSpans( + edits: readonly { start: number; end: number; text: string }[], +): { start: number; end: number }[] { + const sorted = [...edits].sort((a, b) => a.start - b.start || a.end - b.end); + const out: { start: number; end: number }[] = []; + let shift = 0; + for (const e of sorted) { + const start = e.start + shift; + out.push({ start, end: start + e.text.length }); + shift += e.text.length - (e.end - e.start); + } + return out; +} + +/** Text a package file is read as by passes that read package files + * themselves (npm-static): the patched text, or the file. */ +export function packageSourceText(path: string): string | null { + return typePatchedText(path) ?? trackedReadFile(path); +} + +/** The spans a type patch edited in a served file (patched-text offsets), + * for passes that rewrite package JSDoc and must keep patched comments + * (the shape overlay). Empty when nothing was patched. */ +export function typePatchedSpans(fileName: string): readonly { start: number; end: number }[] { + return state?.spans.get(fileName.split("\\").join("/")) ?? []; +} + +/** Every applied match of the current load (for `scriptc types patch + * --list` and tests). */ +export function typePatchMatches(): readonly MatchRecord[] { + return state?.matches ?? []; +} + +function loc(s: State, offset: number): { file: string; start: number; end: number } { + return { file: s.manifest, start: offset, end: offset }; +} + +/** The current load's type patch warnings: SC8030 (no range covers an + * installed copy), SC8031 (a selector matched nothing), SC8032 (redundant: + * the declaration already has the type), SC8033 (ambiguous or not + * patchable as written). Only packages the program loads are reported. */ +export function typePatchDiagnostics(): { + diagnostics: ScrDiagnostic[]; + manifest: string; + manifestText: string; +} | null { + const s = state; + if (s === null) return null; + const out: ScrDiagnostic[] = []; + for (const [root, info] of s.unmatchedCopies) { + const shown = info.version ?? "an unknown version"; + for (const entry of info.entries) { + out.push( + typePatchDiag( + "SC8030", + `type patch '${entry.key}' does not apply to ${info.pkg} ${shown} at ${relativeToManifest(s, root)}; its declarations are used unpatched`, + info.version === null + ? "the installed package has no version" + : `check whether the declarations of ${info.pkg} ${info.version} still need the patch, then widen the range or remove it`, + loc(s, entry.offset), + ), + ); + } + } + for (const r of s.touched) { + const mine = s.matches.filter((m) => m.id === r.id); + // A selector that matched somewhere (the .d.ts) is not a problem where + // it cannot apply (undocumented JavaScript), unless a declaration it + // names was left unpatched. + const problem = s.problems.get(r); + if (problem !== undefined && (mine.length === 0 || problem.always)) { + out.push( + typePatchDiag( + "SC8033", + `type patch '${r.entry.key}': ${problem.message}`, + "name one declaration, or patch the declared type it refers to", + loc(s, r.offset), + ), + ); + continue; + } + if (mine.length === 0) { + out.push( + typePatchDiag( + "SC8031", + `type patch '${r.entry.key}': '${r.selector.text}' matches no declaration in the files of ${r.entry.pkg} this program loads`, + "check the spelling against the package's declarations and JSDoc; the package may have renamed it", + loc(s, r.offset), + ), + ); + continue; + } + if (mine.every((m) => m.redundant)) + out.push( + typePatchDiag( + "SC8032", + `type patch '${r.entry.key}': '${r.selector.text}' is already declared as ${mine[0]!.before}`, + "the package's declarations already say this; remove the patch", + loc(s, r.offset), + ), + ); + } + return { diagnostics: out, manifest: s.manifest, manifestText: s.manifestText }; +} + +function relativeToManifest(s: State, path: string): string { + return relative(dirname(s.manifest), path).split("\\").join("/"); +} diff --git a/packages/compiler/src/frontend/type-patches/locate.test.ts b/packages/compiler/src/frontend/type-patches/locate.test.ts new file mode 100644 index 000000000..d43695d9a --- /dev/null +++ b/packages/compiler/src/frontend/type-patches/locate.test.ts @@ -0,0 +1,201 @@ +import { describe, expect, test } from "vitest"; +import { nodeFrontendServices } from "../services-node.js"; +import { applyPatchEdits, patchFile, type PatchRequest } from "./locate.js"; +import { parseSelector, parseTypeText, type Selector, type TypePatchOp } from "./selector.js"; + +function patch(file: string, text: string, patches: Record) { + const requests: PatchRequest[] = Object.entries(patches).map(([sel, type], id) => ({ + id, + selector: parseSelector(sel) as Selector, + op: parseTypeText(type) as TypePatchOp, + })); + const kind = /\.[cm]?ts$/.test(file) ? "ts" : "js"; + const sf = nodeFrontendServices().parse(`/pkg/${file}`, text, kind); + const result = patchFile(file, text, sf, requests); + const out = applyPatchEdits(text, result.edits); + expect(out.split("\n").length).toBe(text.split("\n").length); + return { ...result, text: out }; +} + +const COMMANDER_LIKE = `class Command { + /** + * @typedef HelpContext + * @type {object} + * @property {boolean} error + * @property {number} helpWidth + * @property {function} write - includes stripColor if needed + * + * @returns {HelpContext} + * @private + */ + _getOutputContext(contextOptions) { + let helpWidth; + return { error: false, write: () => {}, helpWidth }; + } +} +module.exports = { Command }; +`; + +const MS_LIKE = `/** + * @param {String|Number} val + * @param {Object} [options] + * @return {String|Number} + */ +module.exports = function (val, options) { + return parse(val); +}; + +/** + * @param {String} str + * @return {Number} + */ +function parse(str) { + if (str.length > 100) { + return; + } + return 1; +} +`; + +describe("JSDoc", () => { + test("a @typedef property widens", () => { + const r = patch("lib/command.js", COMMANDER_LIKE, { "HelpContext.helpWidth": "| undefined" }); + expect(r.text).toContain("@property {number | undefined} helpWidth"); + expect(r.text).toContain("@returns {HelpContext}"); + expect(r.matches).toMatchObject([ + { line: 6, before: "number", after: "number | undefined", redundant: false }, + ]); + }); + + test("returns of export= and of a file-local function widen", () => { + const r = patch("index.js", MS_LIKE, { + "export=()": "| undefined", + "index.js:parse()": "| undefined", + }); + expect(r.text).toContain("@return {String|Number | undefined}\n */\nmodule.exports"); + expect(r.text).toContain("@return {Number | undefined}\n */\nfunction parse"); + expect(r.problems).toEqual([]); + }); + + test("a file prefix restricts the root to that file", () => { + const r = patch("other.js", MS_LIKE, { "index.js:parse()": "| undefined" }); + expect(r.matches).toEqual([]); + expect(r.text).toBe(MS_LIKE); + }); + + test("parameters, optional parameters and redundancy", () => { + const r = patch("index.js", MS_LIKE, { + "export=(options)": "| undefined", + "parse(str)": "string | undefined", + }); + expect(r.matches[0]).toMatchObject({ redundant: true }); + expect(r.text).toContain("@param {string | undefined} str"); + }); + + test("a replacement on an undocumented declaration inserts a tag", () => { + const text = + "/** Adds. */\nfunction add(a, b) { return a + b; }\nfunction sub(a, b) { return a - b; }\n"; + const r = patch("index.js", text, { "add()": "number", "sub(a)": "number" }); + expect(r.text).toBe( + "/** Adds. @returns {number} */\nfunction add(a, b) { return a + b; }\n/** @param {number} a */ function sub(a, b) { return a - b; }\n", + ); + }); + + test("a widening needs a written type", () => { + const r = patch("index.js", "function f() { return 1; }\n", { "f()": "| undefined" }); + expect(r.problems[0]!.message).toContain("no written type to widen"); + }); + + test("members of exported objects, classes and constructor assignments", () => { + const text = `class Box { + constructor() { + /** @type {string} */ + this.label = ""; + } + /** @returns {number} */ + size() { return 1; } +} +module.exports = { + /** @returns {string} */ + name() { return ""; }, + Box, +}; +`; + const r = patch("index.js", text, { + "Box.label": "| undefined", + "Box.size()": "| null", + "export=.name()": "| undefined", + }); + expect(r.text).toContain("@type {string | undefined} */\n this.label"); + expect(r.text).toContain("@returns {number | null} */\n size()"); + expect(r.text).toContain("@returns {string | undefined} */\n name()"); + }); + + test("function types are parenthesized before widening", () => { + const r = patch("index.js", "/** @type {(a: string) => void} */\nvar cb = null;\n", { + cb: "| undefined", + }); + expect(r.text).toContain("@type {((a: string) => void) | undefined}"); + }); +}); + +describe("declaration files", () => { + const DTS = `export interface Options { + name: string; + retries?: number; + env: Env; +} +export interface Env { + [key: string]: string; +} +export declare function fetchUser(id: string): Promise<{ email: string; id: string }>; +export declare function load(): User; +interface User { name: string; tags: string[] } +declare function ms(value: string): number; +declare function ms(value: number): string; +declare namespace ms { const version: string; } +export = ms; +`; + + test("members, optional members, index signatures and navigation", () => { + const r = patch("index.d.ts", DTS, { + "Options.name": "| undefined", + "Options.retries": "| undefined", + "Env[string]": "| undefined", + "fetchUser().email": "string | null", + "load().tags[]": "| undefined", + "Options.env[string]": "| null", + }); + expect(r.text).toContain(" name: string | undefined;"); + expect(r.text).toContain("[key: string]: string | undefined | null;"); + expect(r.text).toContain("Promise<{ email: string | null; id: string }>"); + expect(r.text).toContain("tags: (string | undefined)[]"); + expect(r.matches.find((m) => m.id === 1)).toMatchObject({ redundant: true }); + }); + + test("export= widens every overload, and a replacement of several is refused", () => { + const r = patch("index.d.ts", DTS, { "export=()": "| undefined" }); + expect(r.text).toContain("declare function ms(value: string): number | undefined;"); + expect(r.text).toContain("declare function ms(value: number): string | undefined;"); + const refused = patch("index.d.ts", DTS, { "export=()": "number" }); + expect(refused.problems[0]!.message).toContain("matches 2 declarations"); + expect(refused.text).toBe(DTS); + }); + + test("namespace members and generic types", () => { + const r = patch("index.d.ts", DTS, { "ms.version": "| undefined" }); + expect(r.text).toContain("const version: string | undefined;"); + const generic = patch( + "index.d.ts", + "export declare function f(): Map;\n", + { "f().a": "| undefined" }, + ); + expect(generic.problems[0]!.message).toContain("cannot follow"); + }); + + test("multi-line types keep their line count when replaced", () => { + const text = "export interface A {\n b: {\n c: number;\n };\n}\n"; + const r = patch("index.d.ts", text, { "A.b": "string" }); + expect(r.text.startsWith("export interface A {\n b: string\n\n;")).toBe(true); + }); +}); diff --git a/packages/compiler/src/frontend/type-patches/locate.ts b/packages/compiler/src/frontend/type-patches/locate.ts new file mode 100644 index 000000000..6c762f5c3 --- /dev/null +++ b/packages/compiler/src/frontend/type-patches/locate.ts @@ -0,0 +1,970 @@ +/* Finds the declarations a type patch selector names in one package file + * and computes the text edits that apply the patch. + * + * Declaration files (and TypeScript sources) are matched on their syntax + * tree: the edit replaces or widens a written type node. JavaScript is + * matched on its JSDoc: the edit replaces or widens the `{...}` of a + * `@typedef`'s `@property`, a `@returns`, a `@param` or a `@type` tag. + * Executable code is never edited, and no edit adds or removes a line + * break, so every later pass and diagnostic keeps its line. */ + +import * as ts from "../ts7/syntax.js"; +import { tokenStartOfNode } from "../ts7/node-functions.js"; +import { + normalizeType, + splitUnion, + type Selector, + type SelectorSegment, + type TypePatchOp, +} from "./selector.js"; + +export interface PatchRequest { + /** Caller's id, echoed in matches and problems. */ + id: number; + selector: Selector; + op: TypePatchOp; +} + +export interface PatchMatch { + id: number; + /** 1-based line of the patched type. */ + line: number; + before: string; + after: string; + /** The declaration already has the patched type: nothing was edited. */ + redundant: boolean; +} + +export interface PatchProblem { + id: number; + message: string; + /** Reported even when the selector matched elsewhere (an ambiguous + * replacement left a declaration unpatched). */ + always?: true; +} + +export interface PatchEdit { + start: number; + end: number; + text: string; +} + +export interface FilePatchResult { + edits: PatchEdit[]; + matches: PatchMatch[]; + problems: PatchProblem[]; +} + +/** A place a type is (or can be) written. */ +interface Slot { + /** The written type's span, or null when there is none. */ + span: { start: number; end: number } | null; + /** The declaration is optional (`x?:`, `[x]`), so it already admits + * undefined. */ + optional: boolean; + /** Where a replacement goes when nothing is written: the inserted text + * is `prefix + type + suffix`. */ + insert: { at: number; prefix: string; suffix: string } | null; + /** A JSDoc slot inside a comment (one-line edits only). */ + jsdoc: boolean; + /** The type is an array element (`T[]`): a union must be wrapped. */ + wrap?: boolean; +} + +type Step = { slots: Slot[] } | { problem: string }; + +/** Type texts that need parentheses before `| T` is appended. */ +function needsParens(text: string): boolean { + return /=>|\?|\bextends\b|^\s*new\b|^\s*function\b|^\s*!/.test(text); +} + +function lineOf(text: string, pos: number): number { + let line = 1; + for (let i = 0; i < pos && i < text.length; i++) if (text.charCodeAt(i) === 10) line++; + return line; +} + +function editsFor( + text: string, + req: PatchRequest, + slot: Slot, +): { edits: PatchEdit[]; match: PatchMatch } | { problem: string } { + const op = req.op; + if (slot.span === null) { + if (op.kind === "widen" || slot.insert === null) + return { + problem: `'${req.selector.text}' has no written type to widen; give the whole type instead`, + }; + const ins = slot.insert; + return { + edits: [{ start: ins.at, end: ins.at, text: ins.prefix + op.text + ins.suffix }], + match: { + id: req.id, + line: lineOf(text, ins.at), + before: "(none)", + after: op.text, + redundant: false, + }, + }; + } + const { start, end } = slot.span; + const original = text.slice(start, end); + const line = lineOf(text, start); + if (op.kind === "widen") { + const present = new Set(splitUnion(original).map(normalizeType)); + if (slot.optional) present.add("undefined"); + const add = op.arms.filter((a) => !present.has(normalizeType(a))); + if (add.length === 0) + return { + edits: [], + match: { id: req.id, line, before: original, after: original, redundant: true }, + }; + // `(() => void) | undefined`; an array element wraps the whole union: + // `(string | undefined)[]`. + const inner = needsParens(original); + const outer = slot.wrap === true; + const suffix = ` | ${add.join(" | ")}`; + const open = (outer ? "(" : "") + (inner ? "(" : ""); + const close = (inner ? ")" : "") + suffix + (outer ? ")" : ""); + const edits: PatchEdit[] = + open === "" + ? [{ start: end, end, text: close }] + : [ + { start, end: start, text: open }, + { start: end, end, text: close }, + ]; + const after = `${open}${original}${close}`; + return { edits, match: { id: req.id, line, before: original, after, redundant: false } }; + } + if (normalizeType(original) === normalizeType(op.text)) + return { + edits: [], + match: { id: req.id, line, before: original, after: original, redundant: true }, + }; + const breaks = original.split("\n").length - 1; + if (breaks > 0 && slot.jsdoc) + return { + problem: `'${req.selector.text}' names a JSDoc type that spans lines; replace it in one line upstream or widen it`, + }; + const replacement = + slot.wrap === true && (splitUnion(op.text).length > 1 || needsParens(op.text)) + ? `(${op.text})` + : op.text; + return { + edits: [{ start, end, text: replacement + "\n".repeat(breaks) }], + match: { id: req.id, line, before: original, after: replacement, redundant: false }, + }; +} + +/** Applies a file's requests. `sf` is the file's syntax tree, parsed as + * TypeScript for declaration files and TypeScript sources and as + * JavaScript otherwise. */ +export function patchFile( + relPath: string, + text: string, + sf: ts.SourceFile, + requests: readonly PatchRequest[], +): FilePatchResult { + const js = !/\.(d\.)?[mc]?ts$/.test(relPath); + const resolver = js ? new JsResolver(text, sf) : new TsResolver(text, sf); + const result: FilePatchResult = { edits: [], matches: [], problems: [] }; + for (const req of requests) { + if (req.selector.file !== null && req.selector.file !== relPath) continue; + const step = resolver.resolve(req.selector); + if ("problem" in step) { + result.problems.push({ id: req.id, message: step.problem }); + continue; + } + if (step.slots.length === 0) continue; + if (req.op.kind === "replace" && step.slots.length > 1) { + result.problems.push({ + id: req.id, + message: `'${req.selector.text}' matches ${step.slots.length} declarations in ${relPath}; a replacement must name one (widen them, or replace the declaration file)`, + always: true, + }); + continue; + } + for (const slot of step.slots) { + const out = editsFor(text, req, slot); + if ("problem" in out) { + result.problems.push({ id: req.id, message: out.problem }); + continue; + } + result.edits.push(...out.edits); + result.matches.push(out.match); + } + } + return result; +} + +/** Applies edits (sorted by position; at one position, in request order). + * An edit overlapping an earlier replacement is dropped. */ +export function applyPatchEdits(text: string, edits: readonly PatchEdit[]): string { + const sorted = edits + .map((e, i) => ({ e, i })) + .sort((a, b) => a.e.start - b.e.start || a.e.end - b.e.end || a.i - b.i) + .map((x) => x.e); + let out = ""; + let at = 0; + for (const e of sorted) { + if (e.start < at) continue; + out += text.slice(at, e.start) + e.text; + at = e.end; + } + return out + text.slice(at); +} + +function nameText(name: ts.Node | undefined): string | null { + if (name === undefined) return null; + if (ts.isIdentifier(name) || ts.isStringLiteral(name) || ts.isPrivateIdentifier(name)) + return name.text; + if (ts.isNumericLiteral(name)) return name.text; + return null; +} + +function hasDefaultModifier(node: ts.Node): boolean { + return (ts.getModifiers(node) ?? []).some((m) => m.kind === ts.SyntaxKind.DefaultKeyword); +} + +/* ── declaration files ─────────────────────────────────────────────── */ + +type TsPos = { node: ts.Node; optional: boolean; element?: boolean }; + +class TsResolver { + /** Top-level declarations by name, ambient module bodies included. */ + private readonly roots = new Map(); + private exportEquals: string | null = null; + private exportDefault: ts.Node[] = []; + + constructor( + private readonly text: string, + sf: ts.SourceFile, + ) { + this.collect(sf.statements.nodes, true); + } + + private collect(statements: readonly ts.Statement[], top: boolean): void { + for (const s of statements) { + if (ts.isExportAssignment(s)) { + if (ts.isIdentifier(s.expression)) { + if (s.isExportEquals) this.exportEquals = s.expression.text; + else this.exportDefault.push(...this.rootsNamed(s.expression.text)); + } + continue; + } + if (ts.isModuleDeclaration(s) && ts.isStringLiteral(s.name)) { + // `declare module "pkg" { ... }`: its statements are the module's. + if (top && s.body !== undefined && ts.isModuleBlock(s.body)) + this.collect(s.body.statements.nodes, false); + continue; + } + if (ts.isVariableStatement(s)) { + for (const d of s.declarationList.declarations.nodes) this.add(nameText(d.name), d); + continue; + } + const named = + ts.isFunctionDeclaration(s) || + ts.isClassDeclaration(s) || + ts.isInterfaceDeclaration(s) || + ts.isTypeAliasDeclaration(s) || + ts.isModuleDeclaration(s) || + ts.isEnumDeclaration(s); + if (!named) continue; + this.add(nameText((s as ts.FunctionDeclaration).name), s); + if ((ts.isFunctionDeclaration(s) || ts.isClassDeclaration(s)) && hasDefaultModifier(s)) + this.exportDefault.push(s); + } + } + + private add(name: string | null, node: ts.Node): void { + if (name === null) return; + const list = this.roots.get(name); + if (list === undefined) this.roots.set(name, [node]); + else list.push(node); + } + + private rootsNamed(name: string): ts.Node[] { + return this.roots.get(name) ?? []; + } + + resolve(sel: Selector): Step { + let start: ts.Node[]; + if (sel.root === "export=") + start = this.exportEquals === null ? [] : this.rootsNamed(this.exportEquals); + else if (sel.root === "default") start = this.exportDefault; + else start = this.rootsNamed(sel.root); + let positions: TsPos[] = start.map((node) => ({ node, optional: false })); + // A position that cannot continue is a problem only when no other + // position of the same name matches (a function merged with a + // namespace, an export and its declaration). + let problem: string | null = null; + for (const seg of sel.segments) { + const next: TsPos[] = []; + for (const pos of positions) { + const step = this.step(pos, seg); + if (typeof step === "string") problem ??= step; + else next.push(...step); + } + positions = next; + if (positions.length === 0) + return problem === null ? { slots: [] } : { problem: `'${sel.text}' ${problem}` }; + } + const slots: Slot[] = []; + for (const pos of positions) { + const slot = this.slotOf(pos); + if (typeof slot === "string") problem ??= slot; + else slots.push(slot); + } + if (slots.length === 0 && problem !== null) return { problem: `'${sel.text}' ${problem}` }; + return { slots }; + } + + private typeSlot(type: ts.TypeNode | undefined, optional: boolean): Slot { + return { + span: type === undefined ? null : { start: tokenStartOfNode(type), end: type.end }, + optional, + insert: null, + jsdoc: false, + }; + } + + private slotOf(pos: TsPos): Slot | string { + const n = pos.node; + if (ts.isTypeNode(n)) return { ...this.typeSlot(n, pos.optional), wrap: pos.element === true }; + if (ts.isPropertySignature(n) || ts.isPropertyDeclaration(n)) + return this.typeSlot( + n.type, + pos.optional || n.postfixToken?.kind === ts.SyntaxKind.QuestionToken, + ); + if ( + ts.isVariableDeclaration(n) || + ts.isTypeAliasDeclaration(n) || + ts.isIndexSignatureDeclaration(n) + ) + return this.typeSlot(n.type, false); + if (ts.isParameter(n)) + return this.typeSlot(n.type, n.questionToken !== undefined || n.initializer !== undefined); + if (ts.isGetAccessor(n)) return this.typeSlot(n.type, false); + if (this.isFunctionLike(n)) + return "names a function; add '()' for its return type or '(name)' for a parameter"; + return "names a declaration without a type of its own; name one of its members"; + } + + private isFunctionLike(n: ts.Node): boolean { + return ( + ts.isFunctionDeclaration(n) || + ts.isMethodSignatureDeclaration(n) || + ts.isMethodDeclaration(n) || + ts.isCallSignatureDeclaration(n) || + ts.isFunctionTypeNode(n) + ); + } + + /** The type a position stands for when a segment navigates into it. */ + private typeOf(n: ts.Node): ts.TypeNode | undefined { + if ( + ts.isPropertySignature(n) || + ts.isPropertyDeclaration(n) || + ts.isVariableDeclaration(n) || + ts.isTypeAliasDeclaration(n) || + ts.isParameter(n) + ) + return n.type; + return undefined; + } + + /** Object-like members a type node holds, following parentheses, unions, + * intersections, `Promise` and references to declarations in this + * file. Null when it passes through a type it cannot follow. */ + private containers(type: ts.TypeNode, depth = 0): ts.Node[] | null { + if (depth > 8) return null; + if (ts.isTypeLiteralNode(type)) return [type]; + if (ts.isParenthesizedTypeNode(type)) return this.containers(type.type, depth + 1); + if (ts.isUnionTypeNode(type) || ts.isIntersectionTypeNode(type)) { + const out: ts.Node[] = []; + for (const arm of type.types.nodes) { + if (arm.kind === ts.SyntaxKind.LiteralType || this.isKeywordType(arm)) continue; + const c = this.containers(arm, depth + 1); + if (c === null) return null; + out.push(...c); + } + return out; + } + if (ts.isTypeReferenceNode(type) && ts.isIdentifier(type.typeName)) { + const name = type.typeName.text; + const args = type.typeArguments?.nodes ?? []; + if ((name === "Promise" || name === "PromiseLike") && args.length === 1) + return this.containers(args[0]!, depth + 1); + if (args.length > 0) return null; + const decls = this.rootsNamed(name); + if (decls.length === 0) return null; + const out: ts.Node[] = []; + for (const d of decls) { + if (ts.isTypeAliasDeclaration(d)) { + const c = this.containers(d.type, depth + 1); + if (c === null) return null; + out.push(...c); + } else out.push(d); + } + return out; + } + return null; + } + + private isKeywordType(n: ts.Node): boolean { + return ( + n.kind === ts.SyntaxKind.NullKeyword || + n.kind === ts.SyntaxKind.UndefinedKeyword || + n.kind === ts.SyntaxKind.StringKeyword || + n.kind === ts.SyntaxKind.NumberKeyword || + n.kind === ts.SyntaxKind.BooleanKeyword + ); + } + + private membersOf(n: ts.Node): readonly ts.Node[] | null { + if (ts.isInterfaceDeclaration(n) || ts.isTypeLiteralNode(n) || ts.isClassDeclaration(n)) + return n.members.nodes; + if (ts.isModuleDeclaration(n)) + return n.body !== undefined && ts.isModuleBlock(n.body) ? n.body.statements.nodes : []; + return null; + } + + /** Expands a position into the object-like nodes that hold members. */ + private holders(pos: TsPos): ts.Node[] | string { + const n = pos.node; + if (this.membersOf(n) !== null) return [n]; + if (ts.isTypeAliasDeclaration(n) || ts.isTypeNode(n) || this.typeOf(n) !== undefined) { + const type = ts.isTypeNode(n) ? n : this.typeOf(n)!; + const c = this.containers(type); + if (c === null) + return "passes through a type it cannot follow (a generic, mapped, conditional or imported type); name the declared type itself"; + return c; + } + if (ts.isFunctionDeclaration(n) || ts.isEnumDeclaration(n)) return []; + return "has a member segment after a declaration without members"; + } + + private signaturesOf(pos: TsPos): ts.Node[] | string { + const n = pos.node; + if (this.isFunctionLike(n)) return [n]; + if (ts.isInterfaceDeclaration(n) || ts.isTypeLiteralNode(n)) + return n.members.nodes.filter((m) => ts.isCallSignatureDeclaration(m)); + const type = ts.isTypeNode(n) ? n : this.typeOf(n); + if (type !== undefined) { + let t: ts.TypeNode = type; + while (ts.isParenthesizedTypeNode(t)) t = t.type; + if (ts.isFunctionTypeNode(t)) return [t]; + const c = this.containers(t); + if (c === null) + return "passes through a type it cannot follow; name the declared type itself"; + return c.flatMap((h) => + ts.isInterfaceDeclaration(h) || ts.isTypeLiteralNode(h) + ? h.members.nodes.filter((m) => ts.isCallSignatureDeclaration(m)) + : [], + ); + } + if (ts.isModuleDeclaration(n) || ts.isClassDeclaration(n)) return []; + return "has '()' after a declaration that is not a function"; + } + + private step(pos: TsPos, seg: SelectorSegment): TsPos[] | string { + switch (seg.kind) { + case "member": { + const holders = this.holders(pos); + if (typeof holders === "string") return holders; + const out: TsPos[] = []; + for (const h of holders) { + const members = this.membersOf(h) ?? []; + for (const m of members) { + if (ts.isVariableStatement(m)) { + for (const d of m.declarationList.declarations.nodes) + if (nameText(d.name) === seg.name) out.push({ node: d, optional: false }); + continue; + } + if (nameText((m as ts.PropertySignatureDeclaration).name) === seg.name) + out.push({ node: m, optional: false }); + } + } + return out; + } + case "return": + case "param": { + const sigs = this.signaturesOf(pos); + if (typeof sigs === "string") return sigs; + const out: TsPos[] = []; + for (const s of sigs) { + const sig = s as ts.FunctionDeclaration; + if (seg.kind === "return") { + if (sig.type !== undefined) out.push({ node: sig.type, optional: false }); + else return "has no written return type"; + } else { + for (const p of sig.parameters.nodes) + if (nameText(p.name) === seg.name) out.push({ node: p, optional: false }); + } + } + return out; + } + case "element": { + let t: ts.TypeNode | undefined = ts.isTypeNode(pos.node) ? pos.node : this.typeOf(pos.node); + while (t !== undefined && ts.isParenthesizedTypeNode(t)) t = t.type; + if (t !== undefined && ts.isArrayTypeNode(t)) + return [{ node: t.elementType, optional: false, element: true }]; + if ( + t !== undefined && + ts.isTypeReferenceNode(t) && + ts.isIdentifier(t.typeName) && + (t.typeName.text === "Array" || t.typeName.text === "ReadonlyArray") && + t.typeArguments?.nodes.length === 1 + ) + return [{ node: t.typeArguments.nodes[0]!, optional: false }]; + return "has '[]' after a type that is not an array"; + } + case "index": { + const holders = this.holders(pos); + if (typeof holders === "string") return holders; + const want = + seg.key === "string" ? ts.SyntaxKind.StringKeyword : ts.SyntaxKind.NumberKeyword; + const out: TsPos[] = []; + for (const h of holders) + for (const m of this.membersOf(h) ?? []) + if (ts.isIndexSignatureDeclaration(m) && m.parameters.nodes[0]?.type?.kind === want) + out.push({ node: m, optional: false }); + return out; + } + } + } +} + +/* ── JSDoc in JavaScript ───────────────────────────────────────────── */ + +interface JsDocTag { + tag: string; + /** Inner span of the `{...}` type, or null. */ + type: { start: number; end: number } | null; + /** Parameter or property name (brackets and defaults removed). */ + name: string | null; + optional: boolean; + /** Offset of the `@`. */ + at: number; +} + +interface JsDocComment { + start: number; + end: number; + tags: JsDocTag[]; +} + +function parseJsDoc(text: string, start: number, end: number): JsDocComment { + const tags: JsDocTag[] = []; + const re = /(^|[\s*])@([A-Za-z]+)/g; + re.lastIndex = start + 3; + const body = text.slice(0, end - 2); + for (let m = re.exec(body); m !== null; m = re.exec(body)) { + const at = m.index + m[1]!.length; + const tag = m[2]!; + let i = at + 1 + tag.length; + while (i < body.length && (body[i] === " " || body[i] === "\t")) i++; + let type: JsDocTag["type"] = null; + if (body[i] === "{") { + let depth = 0; + let j = i; + for (; j < body.length; j++) { + if (body[j] === "{") depth++; + else if (body[j] === "}") { + depth--; + if (depth === 0) break; + } + } + if (j >= body.length) continue; + type = { start: i + 1, end: j }; + // Trim the inner span to the type text. + while (type.start < type.end && /\s/.test(text[type.start]!)) type.start++; + while (type.end > type.start && /\s/.test(text[type.end - 1]!)) type.end--; + i = j + 1; + re.lastIndex = i; + } + let name: string | null = null; + let optional = false; + const rest = /^[ \t]*(\[)?[ \t]*([A-Za-z_$][\w$.]*)/.exec(body.slice(i, i + 200)); + if (rest !== null) { + name = rest[2]!; + optional = rest[1] !== undefined; + } + if (type !== null && text[type.end - 1] === "=") optional = true; + tags.push({ tag, type, name, optional, at }); + } + return { start, end, tags }; +} + +/** The spans of a file's string, template and regular expression literal + * tokens. */ +function literalSpans(sf: ts.SourceFile): { start: number; end: number }[] { + const out: { start: number; end: number }[] = []; + const kinds = new Set([ + ts.SyntaxKind.StringLiteral, + ts.SyntaxKind.NoSubstitutionTemplateLiteral, + ts.SyntaxKind.TemplateHead, + ts.SyntaxKind.TemplateMiddle, + ts.SyntaxKind.TemplateTail, + ts.SyntaxKind.RegularExpressionLiteral, + ]); + const visit = (n: ts.Node): void => { + if (kinds.has(n.kind)) out.push({ start: tokenStartOfNode(n), end: n.end }); + ts.forEachChild(n, visit); + }; + visit(sf); + return out.sort((a, b) => a.start - b.start); +} + +const RETURN_TAGS = new Set(["returns", "return"]); +const PROPERTY_TAGS = new Set(["property", "prop"]); + +/** A declaration in JavaScript a selector can stand on. */ +type JsPos = + | { kind: "decl"; node: ts.Node; owner: ts.Node; fn: ts.Node | null; members: ts.Node | null } + | { kind: "typedef"; comment: JsDocComment; tagIndex: number; prefix: string } + | { kind: "slot"; slot: Slot }; + +class JsResolver { + private readonly comments: JsDocComment[] = []; + private readonly roots = new Map(); + private readonly exportEquals: JsPos[] = []; + private readonly exportDefault: JsPos[] = []; + + constructor( + private readonly text: string, + sf: ts.SourceFile, + ) { + // JSDoc comments, not text inside string, template or regular + // expression literals that looks like one. + const literals = literalSpans(sf); + const re = /\/\*\*(?!\/)[\s\S]*?\*\//g; + for (let m = re.exec(text); m !== null; m = re.exec(text)) { + const inside = literals.find((l) => l.start <= m!.index && m!.index < l.end); + if (inside !== undefined) { + re.lastIndex = inside.end; + continue; + } + this.comments.push(parseJsDoc(text, m.index, m.index + m[0].length)); + } + for (const c of this.comments) + c.tags.forEach((t, i) => { + if ((t.tag === "typedef" || t.tag === "callback") && t.name !== null) + this.add(t.name, { kind: "typedef", comment: c, tagIndex: i, prefix: "" }); + }); + // Declarations first: `exports.name = name` may come before the + // function declaration it exports, which is hoisted. + for (const s of sf.statements.nodes) if (!ts.isExpressionStatement(s)) this.collect(s); + for (const s of sf.statements.nodes) if (ts.isExpressionStatement(s)) this.collect(s); + } + + private add(name: string, pos: JsPos): void { + const list = this.roots.get(name); + if (list === undefined) this.roots.set(name, [pos]); + else list.push(pos); + } + + /** The position of a declaration whose value is `value` (or none). */ + private declPos(node: ts.Node, owner: ts.Node, value: ts.Node | undefined): JsPos { + let fn: ts.Node | null = null; + let members: ts.Node | null = null; + if (ts.isFunctionDeclaration(node) || ts.isMethodDeclaration(node)) fn = node; + else if (ts.isClassDeclaration(node)) members = node; + if (value !== undefined) { + let v = value; + while (ts.isParenthesizedExpression(v)) v = v.expression; + if (ts.isFunctionExpression(v) || ts.isArrowFunction(v)) fn = v; + else if (ts.isClassExpression(v) || ts.isObjectLiteralExpression(v)) members = v; + } + return { kind: "decl", node, owner, fn, members }; + } + + private collect(s: ts.Statement): void { + if (ts.isFunctionDeclaration(s) || ts.isClassDeclaration(s)) { + const pos = this.declPos(s, s, undefined); + const name = nameText(s.name); + if (name !== null) this.add(name, pos); + if (hasDefaultModifier(s)) this.exportDefault.push(pos); + return; + } + if (ts.isVariableStatement(s)) { + const decls = s.declarationList.declarations.nodes; + for (const d of decls) { + const name = nameText(d.name); + if (name !== null) + this.add(name, this.declPos(d, decls.length === 1 ? s : d, d.initializer)); + } + return; + } + if (ts.isExportAssignment(s)) { + if (!s.isExportEquals) this.exportDefault.push(this.declPos(s, s, s.expression)); + return; + } + if (!ts.isExpressionStatement(s)) return; + const e = s.expression; + if (!ts.isBinaryExpression(e) || e.operatorToken.kind !== ts.SyntaxKind.EqualsToken) return; + const target = exportTarget(e.left); + if (target === null) return; + if (target === "export=") { + this.exportEquals.push(this.declPos(s, s, e.right)); + // `module.exports = name`: the named declaration is the export. + if (ts.isIdentifier(e.right)) { + const named = this.roots.get(e.right.text); + if (named !== undefined) this.exportEquals.push(...named); + } + } else if (ts.isIdentifier(e.right) && this.roots.has(e.right.text)) { + // `exports.name = local`: the named declaration is the export (and + // already is under its own name). + if (target !== e.right.text) + for (const pos of [...this.roots.get(e.right.text)!]) + if (pos.kind === "decl") this.add(target, pos); + } else this.add(target, this.declPos(s, s, e.right)); + } + + /** The last JSDoc comment directly before a node's first token. */ + private docOf(owner: ts.Node): JsDocComment | null { + const start = tokenStartOfNode(owner); + for (let i = this.comments.length - 1; i >= 0; i--) { + const c = this.comments[i]!; + if (c.end > start) continue; + return /^\s*$/.test(this.text.slice(c.end, start)) ? c : null; + } + return null; + } + + /** Tags of a comment that belong to the declaration it documents, not + * to a `@typedef` or `@callback` block inside it. */ + private ownTags(c: JsDocComment): JsDocTag[] { + const out: JsDocTag[] = []; + let inBlock = false; + for (const t of c.tags) { + if (t.tag === "typedef" || t.tag === "callback") { + inBlock = true; + continue; + } + if (inBlock && (PROPERTY_TAGS.has(t.tag) || t.tag === "type")) continue; + if (inBlock && t.tag !== "param" && !RETURN_TAGS.has(t.tag)) inBlock = false; + if (inBlock) continue; + out.push(t); + } + return out; + } + + private tagSlot(tag: JsDocTag): Slot { + return { span: tag.type, optional: tag.optional, insert: null, jsdoc: true }; + } + + /** A slot for a tag of the declaration `owner` documents; when it has no + * such tag, a replacement inserts one. */ + private docSlot(owner: ts.Node, tagNames: ReadonlySet, name: string | null): Slot { + const doc = this.docOf(owner); + if (doc !== null) { + const tag = this.ownTags(doc).find( + (t) => tagNames.has(t.tag) && (name === null || t.name === name), + ); + if (tag !== undefined && tag.type !== null) return this.tagSlot(tag); + } + const tagName = [...tagNames][0]!; + const named = name === null ? "" : ` ${name}`; + if (doc !== null) { + const at = doc.end - 2; + const pad = /\s/.test(this.text[at - 1] ?? "") ? "" : " "; + return { + span: null, + optional: false, + insert: { at, prefix: `${pad}@${tagName} {`, suffix: `}${named} ` }, + jsdoc: true, + }; + } + return { + span: null, + optional: false, + insert: { at: tokenStartOfNode(owner), prefix: `/** @${tagName} {`, suffix: `}${named} */ ` }, + jsdoc: true, + }; + } + + resolve(sel: Selector): Step { + let positions: JsPos[]; + if (sel.root === "export=") positions = this.exportEquals; + else if (sel.root === "default") positions = this.exportDefault; + else positions = this.roots.get(sel.root) ?? []; + let problem: string | null = null; + for (const seg of sel.segments) { + const next: JsPos[] = []; + for (const pos of positions) { + const step = this.step(pos, seg); + if (typeof step === "string") problem ??= step; + else next.push(...step); + } + positions = next; + if (positions.length === 0) + return problem === null ? { slots: [] } : { problem: `'${sel.text}' ${problem}` }; + } + const slots: Slot[] = []; + for (const pos of positions) { + const slot = this.slotOf(pos); + if (typeof slot === "string") problem ??= slot; + else if (slot !== null) slots.push(slot); + } + if (slots.length === 0 && problem !== null) return { problem: `'${sel.text}' ${problem}` }; + return { slots }; + } + + private slotOf(pos: JsPos): Slot | string | null { + if (pos.kind === "slot") return pos.slot; + if (pos.kind === "typedef") { + const own = pos.comment.tags[pos.tagIndex]!; + if (pos.prefix === "") { + // A typedef's own type: only when it is not the `object` that + // its @property tags describe. + const typeTag = this.typedefTypeTag(pos); + if (typeTag === null || /^object$/i.test(this.text.slice(typeTag.start, typeTag.end))) + return "names a @typedef of an object; name one of its properties"; + return { span: typeTag, optional: false, insert: null, jsdoc: true }; + } + void own; + return null; + } + if (pos.fn !== null && pos.members === null) + return "names a function; add '()' for its return type or '(name)' for a parameter"; + if (pos.members !== null && !ts.isObjectLiteralExpression(pos.members)) + return "names a class; name one of its members"; + return this.docSlot(pos.owner, new Set(["type"]), null); + } + + private typedefTypeTag(pos: { + comment: JsDocComment; + tagIndex: number; + }): { start: number; end: number } | null { + const tags = pos.comment.tags; + const own = tags[pos.tagIndex]!; + if (own.type !== null) return own.type; + for (let i = pos.tagIndex + 1; i < tags.length; i++) { + const t = tags[i]!; + if (t.tag === "type") return t.type; + if (!PROPERTY_TAGS.has(t.tag)) break; + } + return null; + } + + private step(pos: JsPos, seg: SelectorSegment): JsPos[] | string { + // A JSDoc type is not navigated (a nested @property path continues + // from its typedef position instead). + if (pos.kind === "slot") return []; + if (pos.kind === "typedef") { + const tags = pos.comment.tags; + const own = tags[pos.tagIndex]!; + if (own.tag === "callback") { + if (seg.kind !== "return" && seg.kind !== "param") + return "has a member segment after a @callback"; + for (let i = pos.tagIndex + 1; i < tags.length; i++) { + const t = tags[i]!; + if (t.tag === "typedef" || t.tag === "callback") break; + if ( + seg.kind === "return" + ? RETURN_TAGS.has(t.tag) + : t.tag === "param" && t.name === seg.name + ) + return t.type === null ? [] : [{ kind: "slot", slot: this.tagSlot(t) }]; + } + return []; + } + if (seg.kind !== "member") return "can only name properties of a @typedef"; + const name = pos.prefix === "" ? seg.name : `${pos.prefix}.${seg.name}`; + for (let i = pos.tagIndex + 1; i < tags.length; i++) { + const t = tags[i]!; + if (t.tag === "typedef" || t.tag === "callback") break; + if (!PROPERTY_TAGS.has(t.tag) && t.tag !== "type") break; + if (PROPERTY_TAGS.has(t.tag) && t.name === name && t.type !== null) { + // A nested property path continues from here. + return [ + { kind: "slot", slot: this.tagSlot(t) }, + { ...pos, prefix: name }, + ]; + } + } + return []; + } + switch (seg.kind) { + case "return": + if (pos.fn === null) return "has '()' after a declaration that is not a function"; + return [{ kind: "slot", slot: this.docSlot(this.docOwner(pos), RETURN_TAGS, null) }]; + case "param": { + if (pos.fn === null) return "has '(name)' after a declaration that is not a function"; + const fn = pos.fn as ts.FunctionExpression; + if (!fn.parameters.nodes.some((p) => nameText(p.name) === seg.name)) return []; + return [ + { kind: "slot", slot: this.docSlot(this.docOwner(pos), new Set(["param"]), seg.name) }, + ]; + } + case "member": { + if (pos.members === null) return []; + return this.membersNamed(pos.members, seg.name); + } + default: + return "navigates into a JSDoc type; name the declared type itself"; + } + } + + /** The node whose leading comment documents a declaration. */ + private docOwner(pos: { owner: ts.Node }): ts.Node { + return pos.owner; + } + + private membersNamed(holder: ts.Node, name: string): JsPos[] { + const out: JsPos[] = []; + if (ts.isObjectLiteralExpression(holder)) { + for (const p of holder.properties.nodes) { + if (nameText((p as ts.PropertyAssignment).name) !== name) continue; + if (ts.isPropertyAssignment(p)) out.push(this.declPos(p, p, p.initializer)); + else if (ts.isMethodDeclaration(p)) out.push(this.declPos(p, p, undefined)); + // `{ parse }`: the declaration of that name. + else if (ts.isShorthandPropertyAssignment(p)) out.push(...(this.roots.get(name) ?? [])); + } + return out; + } + const cls = holder as ts.ClassDeclaration; + for (const m of cls.members.nodes) { + if (ts.isConstructorDeclaration(m) && m.body !== undefined) { + // `this.name = value` in the constructor, documented by its comment. + for (const s of m.body.statements.nodes) { + if (!ts.isExpressionStatement(s) || !ts.isBinaryExpression(s.expression)) continue; + const e = s.expression; + if ( + e.operatorToken.kind === ts.SyntaxKind.EqualsToken && + ts.isPropertyAccessExpression(e.left) && + e.left.expression.kind === ts.SyntaxKind.ThisKeyword && + nameText(e.left.name) === name + ) + out.push(this.declPos(s, s, e.right)); + } + continue; + } + if (nameText((m as ts.MethodDeclaration).name) !== name) continue; + if (ts.isMethodDeclaration(m)) out.push(this.declPos(m, m, undefined)); + else if (ts.isPropertyDeclaration(m)) out.push(this.declPos(m, m, m.initializer)); + else if (ts.isGetAccessor(m)) + out.push({ kind: "decl", node: m, owner: m, fn: m, members: null }); + } + return out; + } +} + +/** `module.exports` → "export="; `exports.x` / `module.exports.x` → "x". */ +function exportTarget(left: ts.Expression): string | null { + if (!ts.isPropertyAccessExpression(left)) return null; + const obj = left.expression; + const prop = nameText(left.name); + if (prop === null) return null; + if (ts.isIdentifier(obj) && obj.text === "module" && prop === "exports") return "export="; + if (ts.isIdentifier(obj) && obj.text === "exports") return prop; + if ( + ts.isPropertyAccessExpression(obj) && + ts.isIdentifier(obj.expression) && + obj.expression.text === "module" && + nameText(obj.name) === "exports" + ) + return prop; + return null; +} diff --git a/packages/compiler/src/frontend/type-patches/selector.test.ts b/packages/compiler/src/frontend/type-patches/selector.test.ts new file mode 100644 index 000000000..716c994a8 --- /dev/null +++ b/packages/compiler/src/frontend/type-patches/selector.test.ts @@ -0,0 +1,59 @@ +import { describe, expect, test } from "vitest"; +import { parseSelector, parseTypeText, splitUnion } from "./selector.js"; + +describe("parseSelector", () => { + test("members, returns, parameters, elements and index signatures", () => { + expect(parseSelector("HelpContext.helpWidth")).toEqual({ + file: null, + root: "HelpContext", + segments: [{ kind: "member", name: "helpWidth" }], + text: "HelpContext.helpWidth", + }); + expect(parseSelector("index.js:parse()")).toMatchObject({ + file: "index.js", + root: "parse", + segments: [{ kind: "return" }], + }); + expect(parseSelector("export=()")).toMatchObject({ + root: "export=", + segments: [{ kind: "return" }], + }); + expect(parseSelector("fetchUser(id)")).toMatchObject({ + segments: [{ kind: "param", name: "id" }], + }); + expect(parseSelector("list()[].name")).toMatchObject({ + segments: [{ kind: "return" }, { kind: "element" }, { kind: "member", name: "name" }], + }); + expect(parseSelector("Env[string]")).toMatchObject({ + segments: [{ kind: "index", key: "string" }], + }); + }); + + test("malformed selectors say why", () => { + expect(parseSelector("")).toEqual({ error: "is empty" }); + expect(parseSelector("../x.js:f()")).toEqual({ error: "names a file outside the package" }); + expect(parseSelector("1abc")).toHaveProperty("error"); + expect(parseSelector("a.")).toHaveProperty("error"); + expect(parseSelector("a(1)")).toHaveProperty("error"); + }); +}); + +describe("parseTypeText", () => { + test("a leading bar widens, anything else replaces", () => { + expect(parseTypeText("| undefined")).toEqual({ kind: "widen", arms: ["undefined"] }); + expect(parseTypeText("| undefined | null")).toEqual({ + kind: "widen", + arms: ["undefined", "null"], + }); + expect(parseTypeText("string | null")).toEqual({ kind: "replace", text: "string | null" }); + expect(parseTypeText("a\nb")).toEqual({ error: "must be one line" }); + }); + + test("splitUnion keeps nested unions and arrows together", () => { + expect(splitUnion("(a | b) | (() => c | d) | { x: e | f }")).toEqual([ + "(a | b)", + "(() => c | d)", + "{ x: e | f }", + ]); + }); +}); diff --git a/packages/compiler/src/frontend/type-patches/selector.ts b/packages/compiler/src/frontend/type-patches/selector.ts new file mode 100644 index 000000000..8fdd8b67c --- /dev/null +++ b/packages/compiler/src/frontend/type-patches/selector.ts @@ -0,0 +1,133 @@ +/* Type patch selectors: names of declarations in a package, written the + * same way for JSDoc in its JavaScript and for its declaration files. + * + * selector := [ file ":" ] root segment* + * root := identifier | "export=" | "default" + * segment := "." name member (interface, type alias, @typedef, + * class, namespace, object type) + * | "()" return type + * | "(" name ")" type of the named parameter + * | "[]" element type of an array type + * | "[string]" | "[number]" value type of an index signature + * + * Examples: `HelpContext.helpWidth`, `index.js:parse()`, `export=()`, + * `fetchUser().email`, `Env[string]`. */ + +export type SelectorSegment = + | { kind: "member"; name: string } + | { kind: "return" } + | { kind: "param"; name: string } + | { kind: "element" } + | { kind: "index"; key: "string" | "number" }; + +export interface Selector { + /** Package-relative file the root must be declared in, or null. */ + file: string | null; + /** A declared name, `export=` or `default`. */ + root: string; + segments: readonly SelectorSegment[]; + /** The selector as written. */ + text: string; +} + +const IDENT = /^[A-Za-z_$][\w$]*/; + +/** The parsed selector, or a message saying what is wrong with it. */ +export function parseSelector(text: string): Selector | { error: string } { + let rest = text.trim(); + if (rest === "") return { error: "is empty" }; + let file: string | null = null; + const colon = rest.indexOf(":"); + if (colon >= 0) { + file = rest.slice(0, colon).replace(/^\.\//, ""); + rest = rest.slice(colon + 1); + if (file === "" || file.startsWith("/") || file.split("/").includes("..")) + return { error: "names a file outside the package" }; + } + let root: string; + if (rest.startsWith("export=")) { + root = "export="; + rest = rest.slice("export=".length); + } else { + const m = IDENT.exec(rest); + if (m === null) return { error: "must start with a declared name, 'export=' or 'default'" }; + root = m[0]; + rest = rest.slice(root.length); + } + const segments: SelectorSegment[] = []; + while (rest !== "") { + if (rest.startsWith(".")) { + const m = IDENT.exec(rest.slice(1)); + if (m === null) return { error: `has no member name after '.' at '${rest}'` }; + segments.push({ kind: "member", name: m[0] }); + rest = rest.slice(1 + m[0].length); + } else if (rest.startsWith("()")) { + segments.push({ kind: "return" }); + rest = rest.slice(2); + } else if (rest.startsWith("(")) { + const m = /^\(\s*([A-Za-z_$][\w$]*)\s*\)/.exec(rest); + if (m === null) return { error: `has a malformed parameter at '${rest}'` }; + segments.push({ kind: "param", name: m[1]! }); + rest = rest.slice(m[0].length); + } else if (rest.startsWith("[]")) { + segments.push({ kind: "element" }); + rest = rest.slice(2); + } else if (rest.startsWith("[string]") || rest.startsWith("[number]")) { + segments.push({ kind: "index", key: rest.startsWith("[string]") ? "string" : "number" }); + rest = rest.slice("[string]".length); + } else { + return { error: `has an unexpected '${rest}'` }; + } + } + return { file, root, segments, text: text.trim() }; +} + +/** A widening (`| undefined`) or a replacement type. */ +export type TypePatchOp = + | { kind: "widen"; arms: readonly string[] } + | { kind: "replace"; text: string }; + +/** The operation a patch's type text stands for, or a message. */ +export function parseTypeText(text: string): TypePatchOp | { error: string } { + if (/[\r\n]/.test(text)) return { error: "must be one line" }; + const t = text.trim(); + if (t === "") return { error: "is empty" }; + if (t.startsWith("|")) { + const arms = splitUnion(t.slice(1)); + if (arms.some((a) => a === "")) return { error: "has an empty union member" }; + return { kind: "widen", arms }; + } + return { kind: "replace", text: t }; +} + +/** The top-level union members of a type text, trimmed. */ +export function splitUnion(text: string): string[] { + const out: string[] = []; + let depth = 0; + let quote: string | null = null; + let start = 0; + for (let i = 0; i < text.length; i++) { + const ch = text[i]!; + if (quote !== null) { + if (ch === "\\") i++; + else if (ch === quote) quote = null; + continue; + } + if (ch === '"' || ch === "'" || ch === "`") quote = ch; + else if (ch === "(" || ch === "[" || ch === "{" || ch === "<") depth++; + else if (ch === ")" || ch === "]" || ch === "}" || ch === ">") { + // `=>` is not a closing bracket. + if (!(ch === ">" && text[i - 1] === "=")) depth--; + } else if (ch === "|" && depth === 0) { + out.push(text.slice(start, i).trim()); + start = i + 1; + } + } + out.push(text.slice(start).trim()); + return out; +} + +/** Whitespace-insensitive text of a type for comparisons. */ +export function normalizeType(text: string): string { + return text.replace(/\s+/g, ""); +} diff --git a/packages/compiler/src/frontend/type-patches/semver.test.ts b/packages/compiler/src/frontend/type-patches/semver.test.ts new file mode 100644 index 000000000..83663e3a2 --- /dev/null +++ b/packages/compiler/src/frontend/type-patches/semver.test.ts @@ -0,0 +1,48 @@ +import { describe, expect, test } from "vitest"; +import { isValidRange, satisfies } from "./semver.js"; + +describe("satisfies", () => { + test.each([ + ["15.0.0", "^15.0.0", true], + ["15.4.2", "^15.0.0", true], + ["16.0.0", "^15.0.0", false], + ["0.2.5", "^0.2.3", true], + ["0.3.0", "^0.2.3", false], + ["0.0.3", "^0.0.3", true], + ["0.0.4", "^0.0.3", false], + ["1.2.9", "~1.2.3", true], + ["1.3.0", "~1.2.3", false], + ["1.9.0", "~1", true], + ["2.1.3", "2.1.x", true], + ["2.2.0", "2.1.x", false], + ["2.1.3", "2", true], + ["3.0.0", "2", false], + ["2.1.3", "*", true], + ["1.2.3", "1.2.3", true], + ["1.2.4", "=1.2.3", false], + ["1.5.0", ">=1.2.0 <2.0.0", true], + ["2.0.0", ">=1.2.0 <2.0.0", false], + ["2.0.0", ">= 1.2.0", true], + ["1.3.0", ">1.2", true], + ["1.2.9", ">1.2", false], + ["1.2.9", "<=1.2", true], + ["1.3.0", "<=1.2", false], + ["2.3.9", "1.2.3 - 2.3", true], + ["2.4.0", "1.2.3 - 2.3", false], + ["3.1.0", "^1.0.0 || ^3.0.0", true], + ["2.1.0", "^1.0.0 || ^3.0.0", false], + ["2.0.0-beta.1", "^1.0.0", false], + ["2.0.0-beta.1", "^2.0.0", false], + ["2.0.0-beta.2", "^2.0.0-beta.1", true], + ["2.0.0-alpha", "^2.0.0-beta.1", false], + ["2.1.0-beta.2", "^2.0.0-beta.1", false], + ])("%s satisfies %s: %s", (version, range, expected) => { + expect(satisfies(version, range)).toBe(expected); + }); + + test("malformed input never satisfies", () => { + expect(satisfies("not-a-version", "^1.0.0")).toBe(false); + expect(isValidRange("^banana")).toBe(false); + expect(isValidRange(">=1 <2 || 3.x")).toBe(true); + }); +}); diff --git a/packages/compiler/src/frontend/type-patches/semver.ts b/packages/compiler/src/frontend/type-patches/semver.ts new file mode 100644 index 000000000..86e9d1183 --- /dev/null +++ b/packages/compiler/src/frontend/type-patches/semver.ts @@ -0,0 +1,229 @@ +/* The npm semver range subset type patches key on (`commander@^15.0.0`): + * exact versions, x-ranges, `^`, `~`, comparators, hyphen ranges and `||`. + * A prerelease version satisfies a comparator set only when one of the + * set's comparators names a prerelease of the same major.minor.patch, as + * npm does. */ + +interface Version { + major: number; + minor: number; + patch: number; + pre: readonly (string | number)[]; +} + +interface Comparator { + op: "<" | "<=" | ">" | ">=" | "="; + version: Version; +} + +/** A partial version: missing or wildcard parts are null. */ +interface Partial { + major: number | null; + minor: number | null; + patch: number | null; + pre: readonly (string | number)[]; +} + +const VERSION = /^v?(\d+)\.(\d+)\.(\d+)(?:-([0-9A-Za-z.-]+))?(?:\+[0-9A-Za-z.-]+)?$/; + +function preParts(text: string | undefined): (string | number)[] { + if (text === undefined || text === "") return []; + return text.split(".").map((p) => (/^\d+$/.test(p) ? Number(p) : p)); +} + +/** A full version, or null. */ +export function parseVersion(text: string): Version | null { + const m = VERSION.exec(text.trim()); + if (m === null) return null; + return { major: Number(m[1]), minor: Number(m[2]), patch: Number(m[3]), pre: preParts(m[4]) }; +} + +function parsePartial(text: string): Partial | null { + const m = + /^v?(\d+|[xX*])(?:\.(\d+|[xX*]))?(?:\.(\d+|[xX*]))?(?:-([0-9A-Za-z.-]+))?(?:\+[0-9A-Za-z.-]+)?$/.exec( + text, + ); + if (m === null) return null; + const part = (s: string | undefined): number | null => + s === undefined || s === "x" || s === "X" || s === "*" ? null : Number(s); + const major = part(m[1]); + const minor = major === null ? null : part(m[2]); + const patch = minor === null ? null : part(m[3]); + return { major, minor, patch, pre: patch === null ? [] : preParts(m[4]) }; +} + +function v( + major: number, + minor: number, + patch: number, + pre: readonly (string | number)[] = [], +): Version { + return { major, minor, patch, pre }; +} + +/** `-0` marks the lowest prerelease, so `<2.0.0-0` excludes 2.0.0's + * prereleases as npm's desugaring does. */ +const LOWEST: readonly (string | number)[] = [0]; + +function comparePre(a: readonly (string | number)[], b: readonly (string | number)[]): number { + if (a.length === 0 && b.length === 0) return 0; + if (a.length === 0) return 1; + if (b.length === 0) return -1; + for (let i = 0; i < Math.max(a.length, b.length); i++) { + const x = a[i]; + const y = b[i]; + if (x === undefined) return -1; + if (y === undefined) return 1; + if (x === y) continue; + if (typeof x === "number" && typeof y === "number") return x < y ? -1 : 1; + if (typeof x === "number") return -1; + if (typeof y === "number") return 1; + return x < y ? -1 : 1; + } + return 0; +} + +export function compareVersions(a: Version, b: Version): number { + if (a.major !== b.major) return a.major < b.major ? -1 : 1; + if (a.minor !== b.minor) return a.minor < b.minor ? -1 : 1; + if (a.patch !== b.patch) return a.patch < b.patch ? -1 : 1; + return comparePre(a.pre, b.pre); +} + +/** The comparators a partial with an operator stands for. */ +function desugar(op: string, p: Partial): Comparator[] | null { + const { major, minor, patch } = p; + if (op === "^") { + if (major === null) return []; + const lo = v(major, minor ?? 0, patch ?? 0, p.pre); + let hi: Version; + if (major > 0 || minor === null) hi = v(major + 1, 0, 0, LOWEST); + else if (minor > 0 || patch === null) hi = v(0, minor + 1, 0, LOWEST); + else hi = v(0, 0, patch + 1, LOWEST); + return [ + { op: ">=", version: lo }, + { op: "<", version: hi }, + ]; + } + if (op === "~") { + if (major === null) return []; + const lo = v(major, minor ?? 0, patch ?? 0, p.pre); + const hi = minor === null ? v(major + 1, 0, 0, LOWEST) : v(major, minor + 1, 0, LOWEST); + return [ + { op: ">=", version: lo }, + { op: "<", version: hi }, + ]; + } + if (op === "" || op === "=") { + if (major === null) return []; + if (minor === null) + return [ + { op: ">=", version: v(major, 0, 0) }, + { op: "<", version: v(major + 1, 0, 0, LOWEST) }, + ]; + if (patch === null) + return [ + { op: ">=", version: v(major, minor, 0) }, + { op: "<", version: v(major, minor + 1, 0, LOWEST) }, + ]; + return [{ op: "=", version: v(major, minor, patch, p.pre) }]; + } + if (major === null) + return op === "<" || op === ">" ? [{ op: "<", version: v(0, 0, 0, LOWEST) }] : []; + if (op === ">") { + if (minor === null) return [{ op: ">=", version: v(major + 1, 0, 0) }]; + if (patch === null) return [{ op: ">=", version: v(major, minor + 1, 0) }]; + return [{ op: ">", version: v(major, minor, patch, p.pre) }]; + } + if (op === ">=") return [{ op: ">=", version: v(major, minor ?? 0, patch ?? 0, p.pre) }]; + if (op === "<") { + if (minor === null) return [{ op: "<", version: v(major, 0, 0, LOWEST) }]; + if (patch === null) return [{ op: "<", version: v(major, minor, 0, LOWEST) }]; + return [{ op: "<", version: v(major, minor, patch, p.pre) }]; + } + if (op === "<=") { + if (minor === null) return [{ op: "<", version: v(major + 1, 0, 0, LOWEST) }]; + if (patch === null) return [{ op: "<", version: v(major, minor + 1, 0, LOWEST) }]; + return [{ op: "<=", version: v(major, minor, patch, p.pre) }]; + } + return null; +} + +/** One `||` alternative: a list of comparators, or null when malformed. */ +function parseSet(text: string): Comparator[] | null { + const t = text.trim(); + const hyphen = /^(\S+)\s+-\s+(\S+)$/.exec(t); + if (hyphen !== null) { + const lo = parsePartial(hyphen[1]!); + const hi = parsePartial(hyphen[2]!); + if (lo === null || hi === null) return null; + return [...(desugar(">=", lo) ?? []), ...(desugar("<=", hi) ?? [])]; + } + if (t === "") return []; + const out: Comparator[] = []; + // `>= 1.2.3` may carry a space between the operator and the version. + const tokens = t.replace(/(<=|>=|<|>|=|\^|~)\s+/g, "$1").split(/\s+/); + for (const token of tokens) { + const m = /^(<=|>=|<|>|=|\^|~>?|)(.*)$/.exec(token); + if (m === null) return null; + const op = m[1] === "~>" ? "~" : m[1]!; + const p = parsePartial(m[2]!); + if (p === null) return null; + const cs = desugar(op, p); + if (cs === null) return null; + out.push(...cs); + } + return out; +} + +/** The parsed range, or null when it is not a semver range. */ +export function parseRange(range: string): Comparator[][] | null { + const sets: Comparator[][] = []; + for (const alt of range.split("||")) { + const set = parseSet(alt); + if (set === null) return null; + sets.push(set); + } + return sets; +} + +export function isValidRange(range: string): boolean { + return parseRange(range) !== null; +} + +function test(c: Comparator, version: Version): boolean { + const cmp = compareVersions(version, c.version); + switch (c.op) { + case "<": + return cmp < 0; + case "<=": + return cmp <= 0; + case ">": + return cmp > 0; + case ">=": + return cmp >= 0; + default: + return cmp === 0; + } +} + +/** True when `version` satisfies `range` (false for malformed input). */ +export function satisfies(version: string, range: string): boolean { + const ver = parseVersion(version); + const sets = parseRange(range); + if (ver === null || sets === null) return false; + return sets.some((set) => { + if (!set.every((c) => test(c, ver))) return false; + if (ver.pre.length === 0) return true; + // A prerelease only matches a set that names a prerelease of the same + // version tuple (the LOWEST markers of upper bounds do not count). + return set.some( + (c) => + c.version.pre.length > 0 && + c.version.pre !== LOWEST && + c.version.major === ver.major && + c.version.minor === ver.minor && + c.version.patch === ver.patch, + ); + }); +} diff --git a/packages/compiler/src/index.ts b/packages/compiler/src/index.ts index 5daca2539..63cb7602d 100644 --- a/packages/compiler/src/index.ts +++ b/packages/compiler/src/index.ts @@ -52,7 +52,7 @@ import { InternalCompilerError } from "./errors.js"; import { frontendProcessExecArgv } from "./frontend-process-args.js"; import { type ChildProcess, spawn } from "node:child_process"; import { createHash } from "node:crypto"; -import { existsSync } from "node:fs"; +import { existsSync, readFileSync } from "node:fs"; import { copyFile, mkdir, mkdtemp, readFile, rename, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { basename, dirname, join, resolve } from "node:path"; @@ -121,6 +121,7 @@ import { nativeCodegenDiag, nodeIslandNodeDiag, nodeIslandModuleDiag, + isTypePatchCode, type ScrDiagnostic, } from "./diagnostics/diagnostic.js"; import { libnodeCacheKey, libnodeRecord } from "./node-islands/libnode.js"; @@ -489,12 +490,31 @@ interface CompileWarningSink { sourceTexts: Map | null; } +/** The package.json texts type patch warnings point into, re-read for a + * cache hit's code frames (performance warnings need none). */ +function cachedWarningSources(warnings: readonly ScrDiagnostic[]): Map { + const texts = new Map(); + for (const w of warnings) { + if (!isTypePatchCode(w.code) || texts.has(w.loc.file)) continue; + try { + texts.set(w.loc.file, readFileSync(w.loc.file, "utf8")); + } catch { + // The frame is omitted; the message stands alone. + } + } + return texts; +} + /** The build's performance warnings (SC61xx), stored with its cache entry * so a cache hit repeats them. */ function performanceWarnings(sink: CompileWarningSink | undefined): { performance?: ScrDiagnostic[]; } { - const performance = (sink?.warnings ?? []).filter((d) => isPerformanceCode(d.code)); + // Type patch warnings (SC8030–SC8033) describe the same unchanged inputs, + // so a hit repeats them too. + const performance = (sink?.warnings ?? []).filter( + (d) => isPerformanceCode(d.code) || isTypePatchCode(d.code), + ); return performance.length === 0 ? {} : { performance }; } @@ -1657,7 +1677,7 @@ async function compileTracked( // names is just as untyped as when it was built. if (warningSink !== undefined && earlyHit.performance !== undefined) { warningSink.warnings = earlyHit.performance; - warningSink.sourceTexts = new Map(); + warningSink.sourceTexts = cachedWarningSources(earlyHit.performance); } const executableCacheOptions = earlyCacheOptions; // Route/proof metadata is independently evictable. A full-compiler diff --git a/packages/compiler/src/ir/builtin-effects.ts b/packages/compiler/src/ir/builtin-effects.ts index 5dec23a72..2908b143f 100644 --- a/packages/compiler/src/ir/builtin-effects.ts +++ b/packages/compiler/src/ir/builtin-effects.ts @@ -882,6 +882,7 @@ export const MAY_THROW_LIB_FNS: ReadonlySet = new Set([ "net.setAutoSelTimeout", "error.argTypeThrow", "error.propTypeThrow", + "array.foreignThrow", "emitter.setMaxChk", "emitter.setDefaultMaxChk", "fs.readFileSyncBytes", @@ -1083,6 +1084,7 @@ export const ALWAYS_THROW_LIB_FNS: ReadonlySet = new Set([ "error.nodeThrow", "error.argTypeThrow", "error.propTypeThrow", + "array.foreignThrow", "fs.mkdtempChk", "fs.readFileChk", "fs.opendirChk", diff --git a/packages/compiler/src/ir/builtin-signatures.ts b/packages/compiler/src/ir/builtin-signatures.ts index ec8bd5951..0cfd2076b 100644 --- a/packages/compiler/src/ir/builtin-signatures.ts +++ b/packages/compiler/src/ir/builtin-signatures.ts @@ -857,6 +857,7 @@ function buildLibFnSignatures() { // (subject, characters, NUL-separated replacement texts): the fused // chain of global one-character replacements. "str.replaceChars": { argTypes: [STRING, STRING, STRING], result: STRING }, + "str.codePoints": { argTypes: [STRING], result: arrayOf(STRING) }, /* ToNumber(string) — ECMA-262 7.1.4.1 StringToNumber (scr_string.c): * trim the JS StrWhiteSpace set, empty/whitespace-only → +0, then the * whole span must be one StrNumericLiteral — signed decimal (Infinity @@ -1556,15 +1557,18 @@ function buildLibFnSignatures() { * true } and a callback taking an address string or list, then dials * the answers in order. MAY THROW (host validation, a lookup throw). */ "net.connectLookupDyn": { argTypes: [F64, STRING, DYN], result: NETSOCKET_T }, - "net.sockWrite": { argTypes: [NETSOCKET_T, STRING], result: VOID }, - "net.sockWriteBytes": { argTypes: [NETSOCKET_T, BYTES_U8], result: VOID }, + /** socket.write: Node's Writable answer — false when the writes not yet + * completely sent, each counted in full, reach the 65536-byte + * high-water mark ('drain' follows), and on an ended or closed socket. */ + "net.sockWrite": { argTypes: [NETSOCKET_T, STRING], result: BOOL }, + "net.sockWriteBytes": { argTypes: [NETSOCKET_T, BYTES_U8], result: BOOL }, "net.sockEnd": { argTypes: [NETSOCKET_T], result: VOID }, "net.sockEndStr": { argTypes: [NETSOCKET_T, STRING], result: VOID }, "net.sockEndBytes": { argTypes: [NETSOCKET_T, BYTES_U8], result: VOID }, /** write/end with a CHECKED-DYNAMIC chunk (an untyped JS payload into a * typed socket): the runtime dispatches STR/BYTES and throws Node's * ERR_INVALID_ARG_TYPE chunk TypeError on any other kind. */ - "net.sockWriteDyn": { argTypes: [NETSOCKET_T, DYN], result: VOID }, + "net.sockWriteDyn": { argTypes: [NETSOCKET_T, DYN], result: BOOL }, "net.sockEndDyn": { argTypes: [NETSOCKET_T, DYN], result: VOID }, "net.sockDestroy": { argTypes: [NETSOCKET_T], result: VOID }, "net.sockPipe": { argTypes: [NETSOCKET_T, NETSOCKET_T], result: VOID }, @@ -2022,6 +2026,12 @@ function buildLibFnSignatures() { argTypes: [NETSOCKET_T, { kind: "func", params: [], ret: VOID }, BOOL], result: VOID, }, + /** on('drain') / once('drain'): fires after a write answered false, once + * the write buffer empties. */ + "net.sockOnDrain": { + argTypes: [NETSOCKET_T, { kind: "func", params: [], ret: VOID }, BOOL], + result: VOID, + }, "net.sockRemoteAddress": { argTypes: [NETSOCKET_T], result: VOID }, /** socket.encrypted — `boolean | undefined`: the true arm iff the * socket carries a TLS transport (Node types `encrypted: true` on @@ -2148,6 +2158,12 @@ function buildLibFnSignatures() { /** socket.address(): { address, family, port } or {} as a dyn. */ "net.sockAddress": { argTypes: [NETSOCKET_T], result: DYN }, "net.sockBytesWritten": { argTypes: [NETSOCKET_T], result: F64 }, + /** socket.writableLength: Node's state.length, the length of every + * write not yet completely sent. */ + "net.sockWritableLength": { argTypes: [NETSOCKET_T], result: F64 }, + /** socket.writableNeedDrain: a write answered false and 'drain' has not + * fired, with the write half open. */ + "net.sockWritableNeedDrain": { argTypes: [NETSOCKET_T], result: BOOL }, /** remotePort (0), remoteFamily (1), localAddress (2), localPort (3), * localFamily (4): dyn values, undefined when unknown. */ "net.sockAddrField": { argTypes: [NETSOCKET_T, F64], result: DYN }, @@ -3983,6 +3999,15 @@ function buildLibFnSignatures() { // The string keys `for...in` visits on an array (args[0], any element // type) after its indices. Borrowed; never throws. "array.namedKeys": { argTypes: [null], result: arrayOf(STRING) }, + // The value of the foreign element (SCR_ARR_FOREIGN) at args[1] of an + // array (args[0], any element type); callers test the element state + // first. Borrowed; never throws. + "array.foreign": { argTypes: [null, F64], result: DYN }, + /** The checked conversion's TypeError for the foreign element at args[1] + * of an array (args[0]) read where a value described by args[2] is + * expected. ALWAYS THROWS; polymorphic result (the error.nodeThrow + * pattern). May-throw seed. */ + "array.foreignThrow": { argTypes: [null, F64, STRING], result: VOID }, // Whether the array (args[0]) still has the key (args[1]) from its // namedKeys list. Borrowed; never throws. "array.hasNamedKey": { argTypes: [null, STRING], result: BOOL }, diff --git a/packages/compiler/src/ir/function-references.test.ts b/packages/compiler/src/ir/function-references.test.ts index 970a3da8f..b4f8c51d9 100644 --- a/packages/compiler/src/ir/function-references.test.ts +++ b/packages/compiler/src/ir/function-references.test.ts @@ -221,6 +221,7 @@ const MODULE_FIELDS: Record>, Role> = { "wasmUntranslated[].sha256": "other", "wasmUntranslated[].reason": "other", "threadSlotKeys[]": "other", + "foreignElementKeys[]": "other", "freezableArrays[]": "other", "nodeIsland.libnode": "other", "nodeIsland.nodeVersion": "other", diff --git a/packages/compiler/src/ir/integer-loops.ts b/packages/compiler/src/ir/integer-loops.ts index 7cc49b7a0..ea866a134 100644 --- a/packages/compiler/src/ir/integer-loops.ts +++ b/packages/compiler/src/ir/integer-loops.ts @@ -12,6 +12,9 @@ import { preservesArrayLength } from "./array-length-stability.js"; export interface IntegerBytesForLoop { localId: string; limitReceiver: IrExpr; + /** The receiver is a local of the function that the body never assigns + * (a typed array's length is then the same on every iteration). */ + stableReceiver: boolean; } export interface IntegerCountedForLoop { @@ -682,7 +685,12 @@ export function matchIntegerBytesForLoop( if (!isUnitIncrement(stmt.update, init.localId)) return null; if (writesLocal(stmt.body, init.localId)) return null; - return { localId: init.localId, limitReceiver: cond.right.receiver }; + return { + localId: init.localId, + limitReceiver: cond.right.receiver, + stableReceiver: + receiverLocal !== undefined && !writesLocal(stmt.body, cond.right.receiver.localId), + }; } /** Array lengths are uint32, so zero-based loops and nested `j = i + 1` @@ -729,5 +737,11 @@ export function matchIntegerArrayForLoop( writesLocal(stmt.body, local.id) ) return null; - return { localId: local.id, limitReceiver: cond.right.receiver }; + return { + localId: local.id, + limitReceiver: cond.right.receiver, + stableReceiver: + locals.has(cond.right.receiver.localId) && + !writesLocal(stmt.body, cond.right.receiver.localId), + }; } diff --git a/packages/compiler/src/ir/integer-ranges.ts b/packages/compiler/src/ir/integer-ranges.ts index 1909040c4..51c0a929b 100644 --- a/packages/compiler/src/ir/integer-ranges.ts +++ b/packages/compiler/src/ir/integer-ranges.ts @@ -867,6 +867,8 @@ export function analyzeIntegerRangeFacts( ); if (e.type.kind === "f64") { if (e.method === "length") range = { min: 0, max: STRING_MAX_UNITS }; + // A string holds at most SCR_STR_MAX_CAP (0x7fffffff) bytes. + else if (e.method === "byteLength") range = { min: 0, max: STRING_MAX_UNITS }; // An empty needle is found at the clamped position, up to the length. else if (e.method === "indexOf") range = { min: -1, max: STRING_MAX_UNITS }; } diff --git a/packages/compiler/src/ir/intrinsic-signatures.ts b/packages/compiler/src/ir/intrinsic-signatures.ts index eda569883..3e26e77f2 100644 --- a/packages/compiler/src/ir/intrinsic-signatures.ts +++ b/packages/compiler/src/ir/intrinsic-signatures.ts @@ -31,7 +31,8 @@ export const STR_INTRINSIC_SIGS: Record< normalize: { argTypes: [STRING], minArgs: 1, result: STRING }, isWellFormed: { argTypes: [], minArgs: 0, result: BOOL }, toWellFormed: { argTypes: [], minArgs: 0, result: STRING }, - cpAt: { argTypes: [F64], minArgs: 1, result: STRING }, + byteLength: { argTypes: [], minArgs: 0, result: F64 }, + cpAtByte: { argTypes: [F64], minArgs: 1, result: STRING }, }; /** Per-method signature for regexIntrinsic. test/source/flags take the diff --git a/packages/compiler/src/ir/ir.ts b/packages/compiler/src/ir/ir.ts index 3341ce18a..59ee767f0 100644 --- a/packages/compiler/src/ir/ir.ts +++ b/packages/compiler/src/ir/ir.ts @@ -1079,6 +1079,10 @@ export interface IrModule { * ProgramLifetime instance lives in a counted region (the region.* * intrinsics; backend/llvm/regions.ts). */ lifetimeRegions?: true; + /** Element keys (typeKey of the element type) of native arrays that can + * hold elements outside their element type (SCR_ARR_FOREIGN), stored + * through dynamic views; "*" for every array. Absent when none can. */ + foreignElementKeys?: string[]; /** Classes used as ThreadSlots keys (@scriptc/experimental): the root of * each one's hierarchy carries a slot word (the slots.* intrinsics; * backend/llvm/thread-slots.ts). */ @@ -2471,10 +2475,13 @@ export type IrStrIntrinsicMethod = // surrogate, and the copy with each one replaced by U+FFFD. | "isWellFormed" | "toWellFormed" - // The string iterator's step: the full code-POINT character at a UTF-16 - // index (astral chars come back whole where charAt truncates) — for-of - // over strings desugars onto it, advancing by the result's length. - | "cpAt"; + // The string iterator over the WTF-8 bytes: the byte count of a string, + // and the code-point character that starts at a byte offset (a lone + // surrogate is a one-unit character). for-of, spread and Array.from of a + // string walk a byte cursor with them, advancing by the character's + // byte count; the cursor is always a character start below the count. + | "byteLength" + | "cpAtByte"; /** The typed-array/Buffer method surface (bytesIntrinsic). Receiver/arg * conventions (validated): `length`/`byteLength` are property reads → f64; diff --git a/packages/compiler/src/ir/runtime-features.ts b/packages/compiler/src/ir/runtime-features.ts index e0ecbdc27..3854ab2c8 100644 --- a/packages/compiler/src/ir/runtime-features.ts +++ b/packages/compiler/src/ir/runtime-features.ts @@ -884,10 +884,11 @@ const RUNTIME_FEATURE_RULES = { mod.functions.some( (fn) => fn.async === true && fn.generator === undefined && fn.returnType.kind === "dyn", ), - // The legacy url functions, crypto.createCipheriv and - // ecdh.setPublicKey emit their deprecation warnings through the - // dyn-async unit (WebCrypto's encrypt and decrypt link scr_cipher.c - // too); scr_wasm.c boxes its handles and promises through it. + // The legacy url functions, crypto.createCipheriv, + // ecdh.setPublicKey and generateKeyPair's rsa-pss options emit their + // deprecation warnings through the dyn-async unit (WebCrypto's + // encrypt and decrypt link scr_cipher.c too); scr_wasm.c boxes its + // handles and promises through it. fns: [ ...DYN_ASYNC_LIB_FNS, ...READABLE_DYN_ASYNC_LIB_FNS, @@ -898,6 +899,8 @@ const RUNTIME_FEATURE_RULES = { "webcrypto.wrapKey", "webcrypto.unwrapKey", "crypto.ecdhNew", + "crypto.generateKeyPairCb", + "crypto.generateKeyPairSync", ], prefixes: [...READABLE_DYN_ASYNC_PREFIXES, "wasm."], expr: (node, tables) => diff --git a/packages/compiler/src/ir/validate.ts b/packages/compiler/src/ir/validate.ts index bbb79438d..45ebf5c79 100644 --- a/packages/compiler/src/ir/validate.ts +++ b/packages/compiler/src/ir/validate.ts @@ -4313,7 +4313,9 @@ function validateFunction( if ( e.fn === "array.namedKeys" || e.fn === "array.hasNamedKey" || - e.fn === "array.hasProperties" + e.fn === "array.hasProperties" || + e.fn === "array.foreign" || + e.fn === "array.foreignThrow" ) { if (e.args[0]?.type.kind !== "array") { err(`libCall ${e.fn} arg 0: expected an array, got ${e.args[0]?.type.kind}`, e.loc); diff --git a/packages/compiler/src/project-config.test.ts b/packages/compiler/src/project-config.test.ts index 58ae914ca..04a5d2e0b 100644 --- a/packages/compiler/src/project-config.test.ts +++ b/packages/compiler/src/project-config.test.ts @@ -164,3 +164,80 @@ test("package setting keys named like Object.prototype members are unknown setti ]); expect(c.packages.has("debug")).toBe(false); }); + +test("typePatches: entries keyed by package and range", () => { + const c = config({ + typePatches: { + "commander@^15.0.0": { declarations: { "HelpContext.helpWidth": "| undefined" } }, + "@scope/lib@1.x || 2.x": { + declarations: { "Options.name": "string | null" }, + note: "https://example.invalid/issue/1", + }, + }, + }); + expect(c.diagnostics).toEqual([]); + expect( + c.typePatches.map(({ key, pkg, range, declarations }) => ({ + key, + pkg, + range, + declarations: declarations.map((d) => [d.selector, d.type]), + })), + ).toEqual([ + { + key: "commander@^15.0.0", + pkg: "commander", + range: "^15.0.0", + declarations: [["HelpContext.helpWidth", "| undefined"]], + }, + { + key: "@scope/lib@1.x || 2.x", + pkg: "@scope/lib", + range: "1.x || 2.x", + declarations: [["Options.name", "string | null"]], + }, + ]); + expect(c.typePatches[0]!.declarations[0]!.offset).toBeGreaterThan(0); +}); + +test("typePatches: malformed keys, ranges, selectors and types are SC8004", () => { + const c = config({ + typePatches: { + commander: { declarations: { "A.b": "| undefined" } }, + "ms@^banana": { declarations: {} }, + "@types/node@*": { declarations: {} }, + "typescript@5": { declarations: {} }, + "lib@^1.0.0": { + declarations: { + "1bad": "| undefined", + "A.b": 3, + "A.c": "string\nnumber", + "A.d": "| undefined", + }, + files: {}, + }, + "other@^1.0.0": [], + }, + }); + expect(c.diagnostics.every((d) => d.code === "SC8004")).toBe(true); + expect(c.diagnostics.map((d) => d.message)).toEqual([ + "package.json 'scriptc.typePatches.commander' must name a package and a version range", + `package.json 'scriptc.typePatches.ms@^banana' has the version range "^banana", which is not a semver range`, + "package.json 'scriptc.typePatches.@types/node@*' patches @types/node, whose declarations ship with scriptc or TypeScript", + "package.json 'scriptc.typePatches.typescript@5' patches typescript, whose declarations ship with scriptc or TypeScript", + "package.json 'scriptc.typePatches.lib@^1.0.0.files' is not a known type patch setting", + "package.json 'scriptc.typePatches.lib@^1.0.0.declarations.1bad' is not a selector: it must start with a declared name, 'export=' or 'default'", + "package.json 'scriptc.typePatches.lib@^1.0.0.declarations.A.b' must map to a type string, not 3", + "package.json 'scriptc.typePatches.lib@^1.0.0.declarations.A.c' has a type that must be one line", + "package.json 'scriptc.typePatches.other@^1.0.0' must be an object, not an array", + ]); + // The valid selector of an entry with an unknown key is dropped with it; + // entries with only bad selectors keep their valid ones. + expect(c.typePatches).toEqual([]); + const partial = config({ + typePatches: { + "lib@^1.0.0": { declarations: { "1bad": "| undefined", "A.d": "| undefined" } }, + }, + }); + expect(partial.typePatches[0]!.declarations.map((d) => d.selector)).toEqual(["A.d"]); +}); diff --git a/packages/compiler/src/project-config.ts b/packages/compiler/src/project-config.ts index 997e997c1..c43a6bd2c 100644 --- a/packages/compiler/src/project-config.ts +++ b/packages/compiler/src/project-config.ts @@ -13,7 +13,12 @@ * "proper-lockfile": { "run": "node", "evaluate": "lazy" } * }, * "nodeIslands": { "auto": true, "modules": ["src/plugins/*.ts"] }, - * "wasm": ["build/module.wasm"] + * "wasm": ["build/module.wasm"], + * "typePatches": { // frontend/type-patches/layer.ts + * "commander@^15.0.0": { + * "declarations": { "HelpContext.helpWidth": "| undefined" } + * } + * } * } * * Node islands are report-only in this version: the policy decides which @@ -23,6 +28,8 @@ import { dirname, join, resolve } from "node:path"; import { projectConfigDiag, type ScrDiagnostic } from "./diagnostics/diagnostic.js"; import { trackedReadFile } from "./frontend/input-tracker.js"; +import { parseSelector, parseTypeText } from "./frontend/type-patches/selector.js"; +import { isValidRange } from "./frontend/type-patches/semver.js"; /** How a package runs: compiled natively, in a Node island, or decided by * the detector. */ @@ -57,6 +64,25 @@ export interface NodeIslandsSetting { speculativeStart: boolean; } +/** One declaration patch: a selector and the type text it maps to. */ +export interface TypePatchDeclaration { + selector: string; + type: string; + /** Offset of the selector key in the manifest, for diagnostics. */ + offset: number; +} + +/** One `scriptc.typePatches["@"]` entry. */ +export interface TypePatchEntry { + /** The key as written. */ + key: string; + pkg: string; + range: string; + declarations: TypePatchDeclaration[]; + /** Offset of the key in the manifest. */ + offset: number; +} + export interface ProjectConfig { /** The package.json the configuration came from, or null. */ manifest: string | null; @@ -66,6 +92,8 @@ export interface ProjectConfig { nodeIslands: NodeIslandsSetting; /** `scriptc.wasm`: module paths, resolved against the manifest. */ wasm: string[]; + /** `scriptc.typePatches`, in manifest order. */ + typePatches: TypePatchEntry[]; /** SC8004 for every unknown key or value. */ diagnostics: ScrDiagnostic[]; } @@ -117,6 +145,7 @@ export function emptyProjectConfig(): ProjectConfig { packages: new Map(), nodeIslands: defaultNodeIslands(), wasm: [], + typePatches: [], diagnostics: [], }; } @@ -299,6 +328,9 @@ export function parseProjectConfig(manifest: string, text: string): ProjectConfi } } + const patches = root["typePatches"]; + if (patches !== undefined) readTypePatches(patches, config, problem, locate); + // `scriptc.wasm` keeps its lenient reading: string entries of an array. const wasm = root["wasm"]; if (Array.isArray(wasm)) { @@ -309,6 +341,136 @@ export function parseProjectConfig(manifest: string, text: string): ProjectConfi return config; } +const TYPE_PATCH_KEYS = ["declarations", "note"]; +/** Packages whose declarations ship with scriptc or TypeScript: a wrong + * one is a bug to report, not a project-local patch. */ +const UNPATCHABLE = /^(?:@types\/node|typescript|@typescript\/.*)$/; +const PACKAGE_NAME = /^(?:@[a-z0-9][\w.-]*\/)?[a-z0-9][\w.-]*$/i; + +/** Reads and validates `scriptc.typePatches`. */ +function readTypePatches( + patches: unknown, + config: ProjectConfig, + problem: (path: string[], detail: string, hint: string) => void, + locate: JsonLocator, +): void { + if (!isRecord(patches)) { + problem( + ["typePatches"], + `must be an object mapping "@" to patches, not ${shown(patches)}`, + `write "typePatches": { "@^1.0.0": { "declarations": { "Type.member": "| undefined" } } }`, + ); + return; + } + for (const key of Object.keys(patches)) { + const at = key.lastIndexOf("@"); + const pkg = at > 0 ? key.slice(0, at) : key; + const range = at > 0 ? key.slice(at + 1).trim() : ""; + if (at <= 0 || range === "" || !PACKAGE_NAME.test(pkg)) { + problem( + ["typePatches", key], + "must name a package and a version range", + `write "${at > 0 ? pkg : key}@^1.0.0": the range says which installed versions the patch describes`, + ); + continue; + } + if (!isValidRange(range)) { + problem( + ["typePatches", key], + `has the version range "${range}", which is not a semver range`, + `use a range such as "^${/^\d/.test(range) ? range.replace(/[^\d.]/g, "") : "1.0.0"}" or "1.2.x"`, + ); + continue; + } + if (UNPATCHABLE.test(pkg)) { + problem( + ["typePatches", key], + `patches ${pkg}, whose declarations ship with scriptc or TypeScript`, + "report a wrong declaration there instead of patching it", + ); + continue; + } + const value = patches[key]; + if (!isRecord(value)) { + problem( + ["typePatches", key], + `must be an object, not ${shown(value)}`, + `write { "declarations": { "Type.member": "| undefined" } }`, + ); + continue; + } + const entry: TypePatchEntry = { + key, + pkg, + range, + declarations: [], + offset: locate.offsetOf(["scriptc", "typePatches", key]), + }; + let valid = true; + for (const k of Object.keys(value)) { + if (!TYPE_PATCH_KEYS.includes(k)) { + problem( + ["typePatches", key, k], + "is not a known type patch setting", + `the settings are ${TYPE_PATCH_KEYS.join(", ")}`, + ); + valid = false; + } + } + const note = value["note"]; + if (note !== undefined && typeof note !== "string") { + problem( + ["typePatches", key, "note"], + `must be a string, not ${shown(note)}`, + "write a short note, such as a link to the upstream issue", + ); + valid = false; + } + const decls = value["declarations"]; + if (decls !== undefined && !isRecord(decls)) { + problem( + ["typePatches", key, "declarations"], + `must be an object mapping selectors to types, not ${shown(decls)}`, + `write "declarations": { "Type.member": "| undefined" }`, + ); + valid = false; + } else if (decls !== undefined) { + for (const selector of Object.keys(decls)) { + const path = ["typePatches", key, "declarations", selector]; + const parsed = parseSelector(selector); + if ("error" in parsed) { + problem( + path, + `is not a selector: it ${parsed.error}`, + `name a declaration such as "Type.member", "fn()", "fn(param)", "file.js:local()" or "export=()"`, + ); + continue; + } + const type = decls[selector]; + if (typeof type !== "string") { + problem( + path, + `must map to a type string, not ${shown(type)}`, + `write "| undefined" to widen, or the whole type to replace it`, + ); + continue; + } + const op = parseTypeText(type); + if ("error" in op) { + problem( + path, + `has a type that ${op.error}`, + `write "| undefined" to widen, or the whole type on one line`, + ); + continue; + } + entry.declarations.push({ selector, type, offset: locate.offsetOf(["scriptc", ...path]) }); + } + } + if (valid) config.typePatches.push(entry); + } +} + let cachedManifest: string | null = null; let cachedText: string | null = null; let cachedConfig: ProjectConfig | null = null; diff --git a/packages/compiler/surface-manifest.json b/packages/compiler/surface-manifest.json index 58b3da7e2..5f5d8f7e0 100644 --- a/packages/compiler/surface-manifest.json +++ b/packages/compiler/surface-manifest.json @@ -1,6 +1,6 @@ { "schemaVersion": 2, - "compilerVersion": "0.4.4", + "compilerVersion": "0.4.5", "coverage": [ "Entries are projected mechanically from the compiler's own decision tables: the diagnostics registry, the unsupported-syntax dispatch tables, the stdlib and node-builtin lowering tables, and the supported-builtin-module list. Nothing is hand-maintained; the manifest regenerates byte-identically from the source tree at this version.", "Absence from this manifest means 'not projected', never 'unsupported'. Surfaces lowered through dedicated code paths rather than tables are not yet projected: console, JSON, Promise/async and the timer surface, the net/http/tls/https/dgram/dns/assert/test/stream/readline module member surfaces, template literals, the regex slice, global functions (parseInt, parseFloat, isNaN, isFinite), and the process surface outside its ambient slice.", diff --git a/packages/compiler/test/bytes-header-load-emission.test.ts b/packages/compiler/test/bytes-header-load-emission.test.ts new file mode 100644 index 000000000..bc7cb9803 --- /dev/null +++ b/packages/compiler/test/bytes-header-load-emission.test.ts @@ -0,0 +1,60 @@ +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { expect, test } from "vitest"; +import { compile, deserializeModule, validateModule } from "../src/index.js"; +import { emitLlvmModule } from "../src/backend/llvm/emitter.js"; +import { type IrModule } from "../src/ir/ir.js"; + +async function lower(source: string): Promise { + const dir = await mkdtemp(join(tmpdir(), "scriptc-bytes-header-")); + try { + const entry = join(dir, "main.ts"), + output = join(dir, "main.ir.json"); + await writeFile(entry, source); + const result = await compile(entry, { outDir: dir, outPath: output, outputKind: "ir" }); + if (!result.ok) + throw new Error(result.diagnostics.map((item) => `${item.code}: ${item.message}`).join("\n")); + const module = deserializeModule(await readFile(output, "utf8")); + expect(validateModule(module)).toEqual([]); + return module; + } finally { + await rm(dir, { recursive: true, force: true }); + } +} + +function body(llvm: string, symbol: string): string { + const found = new RegExp(`^define internal [^\\n]*@${symbol}(?:\\.i32)?\\([^]*?^}`, "m").exec( + llvm, + ); + expect(found, symbol).not.toBeNull(); + return found![0]; +} + +const INVARIANT_HEADER = /load (?:i64|i32|ptr), ptr [^\n]*, !tbaa !\d+, !invariant\.load !\{\}$/m; + +test("a typed array's length read as a value is an ordinary load; access bounds stay invariant", async () => { + const module = await lower(` +class Table { + slots: Int32Array = new Int32Array(4); + swap(): number { + const slots = this.slots; + this.slots = new Int32Array(1); + return slots.length + slots.byteLength; + } +} +function sum(values: Float64Array, scale: number): number { + let total = 0; + for (let i = 0; i < values.length; i++) total += values[i]! * scale; + return total; +} +const table = new Table(); +console.log(table.swap(), sum(new Float64Array([1, 2]), 2)); +`); + const llvm = emitLlvmModule(module); + // The length outlives the view: the release must not be crossed. + const swap = body(llvm, "sc_bf__x25_Table_swap"); + expect(swap).toMatch(/load i64, ptr [^\n]*, !tbaa !\d+$/m); + expect(swap).not.toContain("!invariant.load"); + expect(body(llvm, "sc_bf_sum")).toMatch(INVARIANT_HEADER); +}); diff --git a/packages/compiler/test/counted-loop-emission.test.ts b/packages/compiler/test/counted-loop-emission.test.ts index 7a3fc1f44..f617c6ab4 100644 --- a/packages/compiler/test/counted-loop-emission.test.ts +++ b/packages/compiler/test/counted-loop-emission.test.ts @@ -1,4 +1,4 @@ -import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { expect, test } from "vitest"; @@ -45,3 +45,94 @@ test("counted loops guard exact induction and preserve wide remainders on both p await rm(dir, { recursive: true, force: true }); } }); + +test("worker programs poll counted loops over typed arrays between blocks of iterations", async () => { + const dir = await mkdtemp(join(tmpdir(), "scriptc-counted-poll-")); + try { + const source = (workers: boolean): string => + [ + workers ? 'import { isMainThread, Worker } from "node:worker_threads";' : "", + "export function scale(a: Float64Array, b: Float64Array, n: number): void {", + " for (let i = 0; i < n; i++) b[i] = a[i]! * 2;", + "}", + "export function codes(a: Uint8Array): void {", + " for (let i = 0; i < a.length; ++i) a[i] = a[i] === 97 ? 0 : 3;", + "}", + "export function shrink(a: Uint8Array): number {", + " let t = 0;", + " for (let i = 0; i < a.length; ++i) {", + " t += a[i]!;", + " if (i === 2) a = a.subarray(0, 3);", + " }", + " return t;", + "}", + "export function mix(n: number): number {", + " let t = 0;", + " for (let i = 0; i < n; i++) t = (t * 31 + i) % 1000003;", + " return t;", + "}", + "const a = new Float64Array(8).fill(1);", + "const b = new Float64Array(8);", + "scale(a, b, 8);", + "const c = new Uint8Array(8).fill(97);", + "codes(c);", + "console.log(b[7], mix(8), c[0], shrink(new Uint8Array(8).fill(1)));", + workers ? "if (!isMainThread) new Worker(new URL(import.meta.url));" : "", + ].join("\n"); + const emit = async (workers: boolean): Promise => { + const entry = join(dir, `${workers ? "worker" : "plain"}.ts`); + await writeFile(entry, source(workers)); + const outPath = join(dir, `${workers ? "worker" : "plain"}.ir.json`); + const result = await compile(entry, { outPath, outDir: dir, outputKind: "ir" }); + if (!result.ok) throw new Error(result.diagnostics.map((d) => d.message).join("\n")); + return emitLlvmModule(deserializeModule(await readFile(outPath, "utf8"))); + }; + // Every emitted version of a function (the number and int32 entries). + const bodies = (ll: string, name: string): string => + [ + ...ll.matchAll( + new RegExp( + `define internal [^\\n]+ @sc_b?f_${name}(?:\\.i32)?\\([^\\n]*\\n([\\s\\S]*?)\\n\\}`, + "g", + ), + ), + ] + .map((match) => match[1]!) + .join("\n"); + const workerModule = await emit(true); + const worker = bodies(workerModule, "scale"); + const blocks = worker.split(/\n(?=[\w.]+:)/); + // No budget in the body: the inner loop's latch only steps the counter, + // the poll runs after each block, and the block's bound is the nearer of + // the limit and 64 steps on. + expect(worker).not.toContain("worker cancellation budget"); + expect(worker).toContain("@llvm.sadd.sat.i64(i64 %"); + expect(worker).toMatch(/@llvm\.smin\.i64\(i64 %\w+, i64 %\w+\)/); + for (const latch of blocks.filter((block) => block.startsWith("loop.u"))) + expect(latch).not.toContain("scr_exc_flag"); + const polls = blocks.filter((block) => block.startsWith("loop.poll")); + expect(polls.length).toBeGreaterThan(0); + for (const poll of polls) expect(poll).toContain("@scr_exc_flag"); + // A counted loop without typed-array accesses keeps the budget. + // A loop to a typed array's length is strip-mined the same way, with the + // length as each block's limit. + const codes = bodies(workerModule, "codes"); + expect(codes).not.toContain("worker cancellation budget"); + expect(codes).toMatch(/@llvm\.uadd\.sat\.i64\(i64 %\w+, i64 64\)/); + expect(codes).toMatch(/@llvm\.umin\.i64\(i64 %\w+, i64 %\w+\)/); + expect(codes).toMatch(/^loop\.poll\d+:$/m); + // A loop that assigns its array reads the new length on the next + // iteration: it keeps the budget. + const shrink = bodies(workerModule, "shrink"); + expect(shrink).toContain("worker cancellation budget"); + expect(shrink).not.toMatch(/^loop\.poll\d+:$/m); + const mix = bodies(workerModule, "mix"); + expect(mix).toContain("worker cancellation budget"); + expect(mix).not.toContain("loop.poll"); + const plain = bodies(await emit(false), "scale"); + expect(plain).not.toContain("loop.poll"); + expect(plain).not.toContain("@scr_exc_flag"); + } finally { + await rm(dir, { recursive: true, force: true }); + } +}); diff --git a/packages/compiler/test/js-ic-emission.test.ts b/packages/compiler/test/js-ic-emission.test.ts index 79342e99d..b350c743d 100644 --- a/packages/compiler/test/js-ic-emission.test.ts +++ b/packages/compiler/test/js-ic-emission.test.ts @@ -201,7 +201,7 @@ console.log(lib.total([1, 2, 3])); expect(fn).toContain("@scr_js_keyed_probe("); // The length read answers from the native array's length. expect(fn).toContain("icg.rest"); - expect(fn).toContain("@scr_js_get_probe("); + expect(fn).toContain("@scr_js_get_probe_native("); } finally { await rm(dir, { recursive: true, force: true }); } diff --git a/packages/compiler/test/native-codegen-integration.test.ts b/packages/compiler/test/native-codegen-integration.test.ts index dc3f94e6c..a57d96f01 100644 --- a/packages/compiler/test/native-codegen-integration.test.ts +++ b/packages/compiler/test/native-codegen-integration.test.ts @@ -378,7 +378,7 @@ done: expect(await symbols(helperObject, ["-u"])).toEqual(await symbols(clangObject, ["-u"])); expect(await symbols(helperObject, ["-gU"])).toEqual(await symbols(clangObject, ["-gU"])); expect(await symbols(helperObject, ["-gU"])).toEqual(["0000000000000000 T _main"]); - expect(await symbols(helperObject, ["-u"])).toContain("_scr_runtime_abi_v44"); + expect(await symbols(helperObject, ["-u"])).toContain("_scr_runtime_abi_v45"); const clangExe = join(dir, "clang-program"); const linkDriver = join(dir, "clang-driver.c"); @@ -712,7 +712,7 @@ done: .split("\n"); expect(undefinedSymbols).toContain("_sf_scale"); expect(undefinedSymbols).toContain("_sf_callback_mix"); - expect(undefinedSymbols).toContain("_scr_runtime_abi_v44"); + expect(undefinedSymbols).toContain("_scr_runtime_abi_v45"); }); test("helper and clang object paths retain the same outbound FFI C ABI", async () => { diff --git a/packages/compiler/test/regex-replacers.test.ts b/packages/compiler/test/regex-replacers.test.ts new file mode 100644 index 000000000..746279eb4 --- /dev/null +++ b/packages/compiler/test/regex-replacers.test.ts @@ -0,0 +1,42 @@ +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { expect, test } from "vitest"; +import { compile } from "../src/index.js"; + +async function emitLlvm(source: string): Promise { + const dir = await mkdtemp(join(tmpdir(), "scriptc-regex-replacers-")); + try { + const entry = join(dir, "main.ts"); + const outPath = join(dir, "main.ll"); + await writeFile(entry, source); + const result = await compile(entry, { outDir: dir, outPath, outputKind: "llvm" }); + expect(result.ok, JSON.stringify(result.diagnostics)).toBe(true); + return await readFile(outPath, "utf8"); + } finally { + await rm(dir, { recursive: true, force: true }); + } +} + +test("global substitutions of regex literals with literal replacements run a replacer", async () => { + const llvm = await emitLlvm(` +const s = process.argv[2] ?? "ac"; +const rep = process.argv[3] ?? "-"; +console.log( + s.replace(/<[^>]*>/g, "|"), + s.replaceAll(/a[bc]/g, "<2>"), + s.replace(/<[^>]*>/g, "|"), + s.replace(/<[^>]*>/g, "$&$&"), + s.replace(/<[^>]*>/, "|"), + s.replace(/<[^>]*>/gy, "|"), + s.replace(/<[^>]*>/g, rep), +); +`); + const replacers = llvm.match(/^define internal i32 @sc_rr_\d+\(/gm) ?? []; + // One replacer per (pattern, flags, replacement): the repeated site shares it. + expect(replacers).toHaveLength(2); + expect(llvm.match(/call ptr @scr_regex_replace_with\(/g)).toHaveLength(3); + // $ patterns, non-global and sticky patterns, and computed replacements + // keep the runtime's loop. + expect(llvm.match(/call ptr @scr_regex_replace\(/g)!.length).toBeGreaterThanOrEqual(4); +}); diff --git a/packages/compiler/test/runtime-optional-precision.test.ts b/packages/compiler/test/runtime-optional-precision.test.ts index b47cb828d..e8b794b44 100644 --- a/packages/compiler/test/runtime-optional-precision.test.ts +++ b/packages/compiler/test/runtime-optional-precision.test.ts @@ -102,6 +102,82 @@ console.log(copy.map((item) => weigh(item)).join(",")); expect(signature(spread, "weigh")).toBe("weigh(Item|undefined): f64"); }); +test("holes stored through a dynamic view widen callbacks over the arrays it views", async () => { + const viewed = await lower(`${ITEM} +const items: Item[] = [new Item(1)]; +const view: unknown[] = items; +view[3] = new Item(2); +console.log(items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(viewed, "weigh")).toBe("weigh(Item|undefined): f64"); + // A dynamic store cannot reach an element type no dynamic value views. + const other = await lower(`${ITEM} +const items: Item[] = [new Item(1)]; +const values: unknown[] = [1]; +values[3] = 2; +console.log(items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(other, "weigh")).toBe("weigh(Item): f64"); + // A view without stores that leave holes keeps the parameters. + const read = await lower(`${ITEM} +const items: Item[] = [new Item(1)]; +const view: unknown[] = items; +console.log(view.length, items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(read, "weigh")).toBe("weigh(Item): f64"); + // Standard library functions that take arrays as `any` only read them. + const readers = await lower(`${ITEM} +const items: Item[] = [new Item(1)]; +console.log(items.length, JSON.stringify(items)); +const values: unknown[] = [1]; +values[3] = 2; +console.log(items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(readers, "weigh")).toBe("weigh(Item): f64"); + // A store through a Record type is not a store through a view. + const record = await lower(`${ITEM} +const items: Item[] = [new Item(1)]; +const view: unknown = items; +const bag: Record = {}; +bag["3"] = view; +console.log(items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(record, "weigh")).toBe("weigh(Item): f64"); + // An unknown[] that only holds the array its declaration creates is never + // a view, so stores past its end leave the viewed arrays alone. + const fresh = await lower(`${ITEM} +class Slots { + readonly states: unknown[] = []; +} +const slots = new Slots(); +const items: Item[] = [new Item(1)]; +const view: unknown = items; +const states = slots.states; +states[3] = view; +console.log(items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(fresh, "weigh")).toBe("weigh(Item): f64"); + // A mutable property every assignment of which is a fresh array is fresh + // storage too. + const reassigned = await lower(`${ITEM} +class Facts { + private facts: unknown[] = []; + begin(): void { + this.facts = []; + } + add(fact: unknown): void { + this.facts[3] = fact; + } +} +const items: Item[] = [new Item(1)]; +const facts = new Facts(); +facts.begin(); +facts.add(items); +console.log(items.findIndex((item) => weigh(item) > 1)); +`); + expect(signature(reassigned, "weigh")).toBe("weigh(Item): f64"); +}); + test("holes that only their owning slot can observe stay out of callbacks", async () => { const module = await lower(`${ITEM} class Cache { diff --git a/packages/compiler/test/string-iteration-emission.test.ts b/packages/compiler/test/string-iteration-emission.test.ts new file mode 100644 index 000000000..7cda3da55 --- /dev/null +++ b/packages/compiler/test/string-iteration-emission.test.ts @@ -0,0 +1,47 @@ +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, test } from "vitest"; +import { compile } from "../src/index.js"; + +const directories: string[] = []; +afterEach(async () => { + await Promise.all(directories.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); +}); + +async function emit(source: string): Promise { + const outDir = await mkdtemp(join(tmpdir(), "scriptc-string-iteration-")); + directories.push(outDir); + const entry = join(outDir, "main.ts"); + await writeFile(entry, source); + const outPath = join(outDir, "program.ll"); + const result = await compile(entry, { entry, outDir, outPath, outputKind: "llvm" }); + expect(result.ok, JSON.stringify(result)).toBe(true); + return readFile(outPath, "utf8"); +} + +const calls = (llvm: string, symbol: string): string[] => + llvm.split("\n").filter((line) => line.includes("call ") && line.includes(`@${symbol}(`)); + +test("for-of over a string walks a byte cursor", async () => { + const llvm = await emit(` +function count(s: string): number { + let n = 0; + for (const c of s) n += c.length; + return n; +} +console.log(count(process.argv[1] ?? "")); +`); + expect(calls(llvm, "sc_str_cp_at_byte")).toHaveLength(1); + expect(calls(llvm, "scr_str_cp_at")).toHaveLength(0); + // The helper answers ASCII bytes from the one-character table. + expect(llvm).toContain("define internal ptr @sc_str_cp_at_byte(ptr %s, i64 %b)"); +}); + +test("spread and Array.from of a string build the array in one call", async () => { + const llvm = await emit(` +const s = process.argv[1] ?? ""; +console.log([...s].length, Array.from(s).length); +`); + expect(calls(llvm, "scr_str_code_points")).toHaveLength(2); +}); diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6067-view-holes-in-typed-arrays.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6067-view-holes-in-typed-arrays.ts.json new file mode 100644 index 000000000..9d257fd49 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6067-view-holes-in-typed-arrays.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6067-view-holes-in-typed-arrays.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6068-foreign-elements-through-views.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6068-foreign-elements-through-views.ts.json new file mode 100644 index 000000000..7da41a80f --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6068-foreign-elements-through-views.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6068-foreign-elements-through-views.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6069-foreign-element-writers.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6069-foreign-element-writers.ts.json new file mode 100644 index 000000000..973bdb93e --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/6069-foreign-element-writers.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/6069-foreign-element-writers.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/7515-net-socket-backpressure.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/7515-net-socket-backpressure.ts.json new file mode 100644 index 000000000..16f6edb8d --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/7515-net-socket-backpressure.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/7515-net-socket-backpressure.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/7519-net-socket-drain-listener-close.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/7519-net-socket-drain-listener-close.ts.json new file mode 100644 index 000000000..a3c10da99 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/7519-net-socket-drain-listener-close.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/7519-net-socket-drain-listener-close.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8010-http-header-raw-bytes.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8010-http-header-raw-bytes.ts.json new file mode 100644 index 000000000..78240d310 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8010-http-header-raw-bytes.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8010-http-header-raw-bytes.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8011-foreign-element-tojson.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8011-foreign-element-tojson.ts.json new file mode 100644 index 000000000..e11c1b240 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8011-foreign-element-tojson.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8011-foreign-element-tojson.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8165-rsa-keygen-lengths.cjs.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8165-rsa-keygen-lengths.cjs.json new file mode 100644 index 000000000..45346fc86 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8165-rsa-keygen-lengths.cjs.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8165-rsa-keygen-lengths.cjs" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8166-rsa-pss-keys.cjs.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8166-rsa-pss-keys.cjs.json new file mode 100644 index 000000000..623419ef3 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8166-rsa-pss-keys.cjs.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8166-rsa-pss-keys.cjs" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8167-encrypted-private-keys.cjs.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8167-encrypted-private-keys.cjs.json new file mode 100644 index 000000000..e5e03f8e9 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8167-encrypted-private-keys.cjs.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8167-encrypted-private-keys.cjs" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8178-rsa-pss-auto-min-salt.cjs.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8178-rsa-pss-auto-min-salt.cjs.json new file mode 100644 index 000000000..7b3e5eef7 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8178-rsa-pss-auto-min-salt.cjs.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8178-rsa-pss-auto-min-salt.cjs" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8183-key-passphrase-encoding.cjs.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8183-key-passphrase-encoding.cjs.json new file mode 100644 index 000000000..ee89a2add --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8183-key-passphrase-encoding.cjs.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8183-key-passphrase-encoding.cjs" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8240-asserted-callee-errors.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8240-asserted-callee-errors.ts.json new file mode 100644 index 000000000..365bf7076 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8240-asserted-callee-errors.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8240-asserted-callee-errors.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8347-string-iteration-code-points.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8347-string-iteration-code-points.ts.json new file mode 100644 index 000000000..1e83f7b6f --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8347-string-iteration-code-points.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8347-string-iteration-code-points.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8348-string-iteration-worker-characters.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8348-string-iteration-worker-characters.ts.json new file mode 100644 index 000000000..e767eaf65 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8348-string-iteration-worker-characters.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8348-string-iteration-worker-characters.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8351-regex-literal-replacers.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8351-regex-literal-replacers.ts.json new file mode 100644 index 000000000..be0149287 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/8351-regex-literal-replacers.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/8351-regex-literal-replacers.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/intersection-restated-members.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/intersection-restated-members.ts.json new file mode 100644 index 000000000..4560cb4ae --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/intersection-restated-members.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/intersection-restated-members.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/native-array-dynamic-reads/main.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/native-array-dynamic-reads/main.ts.json new file mode 100644 index 000000000..c3b3013ea --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/native-array-dynamic-reads/main.ts.json @@ -0,0 +1,7 @@ +{ + "order": [ + "/tests/corpus/native-array-dynamic-reads/reads.js", + "/tests/corpus/native-array-dynamic-reads/main.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/native-worker-length-loop-reassign.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/native-worker-length-loop-reassign.ts.json new file mode 100644 index 000000000..f5230bace --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/native-worker-length-loop-reassign.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/native-worker-length-loop-reassign.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/native-worker-terminate-counted-loops.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/native-worker-terminate-counted-loops.ts.json new file mode 100644 index 000000000..5f7c54075 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/native-worker-terminate-counted-loops.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/native-worker-terminate-counted-loops.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/typed-array-length-after-release.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/typed-array-length-after-release.ts.json new file mode 100644 index 000000000..ca6fcb551 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/corpus/typed-array-length-after-release.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/corpus/typed-array-length-after-release.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/fixtures/npm/cases/type-patch-dts/main.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/fixtures/npm/cases/type-patch-dts/main.ts.json new file mode 100644 index 000000000..c9a8e15fc --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/fixtures/npm/cases/type-patch-dts/main.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/fixtures/npm/cases/type-patch-dts/main.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/ts7/baselines/order-parity/tests/fixtures/npm/cases/type-patch-types-twin/main.ts.json b/packages/compiler/test/ts7/baselines/order-parity/tests/fixtures/npm/cases/type-patch-types-twin/main.ts.json new file mode 100644 index 000000000..faa48f074 --- /dev/null +++ b/packages/compiler/test/ts7/baselines/order-parity/tests/fixtures/npm/cases/type-patch-types-twin/main.ts.json @@ -0,0 +1,6 @@ +{ + "order": [ + "/tests/fixtures/npm/cases/type-patch-types-twin/main.ts" + ], + "diags": [] +} diff --git a/packages/compiler/test/type-patches.test.ts b/packages/compiler/test/type-patches.test.ts new file mode 100644 index 000000000..b4c0af9bc --- /dev/null +++ b/packages/compiler/test/type-patches.test.ts @@ -0,0 +1,219 @@ +/* The type patch layer (frontend/type-patches/layer.ts) against real + * packages: commander's HelpContext.helpWidth @typedef property and ms's + * @return tags, patched from the fixtures' package.json; installed-version + * selection; and the warnings for patches that do not apply as written. */ +import { mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, test } from "vitest"; +import { nodeFrontendServices } from "../src/frontend/services-node.js"; +import { + configureTypePatches, + typePatchDiagnostics, + typePatchedSpans, + typePatchedText, + typePatchMatches, +} from "../src/frontend/type-patches/layer.js"; + +const fixtures = join(import.meta.dirname, "../../../tests/fixtures"); +const parse = (path: string, text: string, kind: "ts" | "js") => + nodeFrontendServices().parse(path, text, kind); + +afterEach(() => configureTypePatches(join(tmpdir(), "none", "main.ts"), parse)); + +test("commander's HelpContext.helpWidth typedef property reads as number | undefined", () => { + const root = join(fixtures, "commander-calc"); + configureTypePatches(join(root, "calc.ts"), parse); + const command = join(root, "node_modules/commander/lib/command.js"); + const text = typePatchedText(command); + expect(text).toBeDefined(); + expect(text).toContain(" * @property {number | undefined} helpWidth\n"); + expect(text).toContain(" * @returns {HelpContext}\n"); + expect(typePatchMatches()).toMatchObject([ + { + file: command, + line: 2481, + before: "number", + after: "number | undefined", + pkg: "commander", + version: "15.0.0", + }, + ]); + const [span] = typePatchedSpans(command); + expect(text!.slice(span!.start, span!.end)).toBe(" | undefined"); + // Its declaration file has an unrelated HelpContext without helpWidth. + expect(typePatchedText(join(root, "node_modules/commander/typings/index.d.ts"))).toBeUndefined(); + // Other files of the package are served as they are. + expect(typePatchedText(join(root, "node_modules/commander/lib/help.js"))).toBeUndefined(); + expect(typePatchDiagnostics()?.diagnostics).toEqual([]); +}); + +test("ms's parse() and export return tags widen", () => { + const root = join(fixtures, "type-patches"); + configureTypePatches(join(root, "ms.ts"), parse); + const file = join(root, "node_modules/ms/index.js"); + const text = typePatchedText(file)!; + const original = readFileSync(file, "utf8"); + expect(text).toBe( + original + .replace(" * @return {String|Number}\n", " * @return {String|Number | undefined}\n") + .replace( + " * @return {Number}\n * @api private\n */\n\nfunction parse", + " * @return {Number | undefined}\n * @api private\n */\n\nfunction parse", + ), + ); + expect(text).not.toBe(original); + expect(typePatchDiagnostics()?.diagnostics.map((d) => d.code)).toEqual([]); +}); + +test("a project without type patches serves every file unchanged", () => { + configureTypePatches(join(fixtures, "npm-static", "ms-cli.ts"), parse); + expect(typePatchedText(join(fixtures, "npm-static/node_modules/ms/index.js"))).toBeUndefined(); + expect(typePatchDiagnostics()).toBeNull(); +}); + +test("patches that do not apply as written warn", () => { + const root = join(fixtures, "type-patches"); + configureTypePatches(join(root, "stale", "main.ts"), parse); + expect(typePatchedText(join(root, "node_modules/termctx/index.js"))).toBeUndefined(); + typePatchedText(join(root, "node_modules/ms/index.js")); + const diags = typePatchDiagnostics()!.diagnostics; + expect(diags.map((d) => [d.code, d.message])).toEqual([ + [ + "SC8030", + "type patch 'termctx@^2.0.0' does not apply to termctx 1.2.0 at ../node_modules/termctx; its declarations are used unpatched", + ], + [ + "SC8031", + "type patch 'ms@^2.1.0': 'index.js:format()' matches no declaration in the files of ms this program loads", + ], + ["SC8032", "type patch 'ms@^2.1.0': 'index.js:fmtShort(ms)' is already declared as Number"], + [ + "SC8033", + "type patch 'ms@^2.1.0': 'index.js:plural()' has no written type to widen; give the whole type instead (index.js)", + ], + ]); + expect( + diags.every((d) => d.loc.file === join(root, "stale", "package.json") && d.loc.start > 0), + ).toBe(true); +}); + +test("an @types twin is matched by the runtime package's installed version", () => { + const root = mkdtempSync(join(tmpdir(), "scriptc-type-patches-")); + try { + const twin = join(fixtures, "npm/cases/type-patch-types-twin"); + configureTypePatches(join(twin, "main.ts"), parse); + const text = typePatchedText(join(twin, "node_modules/@types/tallyjs/index.d.ts"))!; + expect(text).toContain(" max: number | undefined;\n mean: number | null;\n"); + // A range the installed runtime (0.4.2) does not satisfy leaves it alone, + // even though the twin's own version (0.4.0) would. + writeFileSync( + join(root, "package.json"), + JSON.stringify({ + scriptc: { + typePatches: { "tallyjs@0.4.0": { declarations: { "Totals.max": "| undefined" } } }, + }, + }), + ); + configureTypePatches(join(root, "main.ts"), parse); + expect(typePatchedText(join(twin, "node_modules/@types/tallyjs/index.d.ts"))).toBeUndefined(); + expect(typePatchDiagnostics()!.diagnostics.map((d) => d.code)).toEqual(["SC8030"]); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("an @types twin in pnpm's layout is matched by the runtime package the project resolves", () => { + // pnpm keeps each package in its own directory under node_modules/.pnpm + // and links the project's dependencies from node_modules: the twin has no + // runtime package beside it. + const root = mkdtempSync(join(tmpdir(), "scriptc-type-patches-pnpm-")); + try { + const store = join(root, "node_modules/.pnpm"); + const runtime = join(store, "tallyjs@0.5.0/node_modules/tallyjs"); + const types = join(store, "@types+tallyjs@0.4.0/node_modules/@types/tallyjs"); + mkdirSync(runtime, { recursive: true }); + mkdirSync(types, { recursive: true }); + writeFileSync(join(runtime, "package.json"), '{ "name": "tallyjs", "version": "0.5.0" }'); + writeFileSync(join(runtime, "index.js"), "exports.totals = () => ({});\n"); + writeFileSync(join(types, "package.json"), '{ "name": "@types/tallyjs", "version": "0.4.0" }'); + writeFileSync( + join(types, "index.d.ts"), + "export type Totals = {\n max: number;\n};\nexport declare function totals(): Totals;\n", + ); + mkdirSync(join(root, "node_modules/@types"), { recursive: true }); + symlinkSync(runtime, join(root, "node_modules/tallyjs")); + symlinkSync(types, join(root, "node_modules/@types/tallyjs")); + const patchFor = (range: string) => + writeFileSync( + join(root, "package.json"), + JSON.stringify({ + scriptc: { + typePatches: { + [`tallyjs@${range}`]: { declarations: { "Totals.max": "| undefined" } }, + }, + }, + }), + ); + // A range the runtime (0.5.0) satisfies applies, though the twin's own + // version (0.4.0) does not satisfy it. + patchFor("0.5.0"); + configureTypePatches(join(root, "main.ts"), parse); + expect(typePatchedText(join(types, "index.d.ts"))).toContain("max: number | undefined;"); + // A range only the twin's own version satisfies does not. + patchFor("0.4.0"); + configureTypePatches(join(root, "main.ts"), parse); + expect(typePatchedText(join(types, "index.d.ts"))).toBeUndefined(); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("an export before its hoisted function is patched, and string text that looks like JSDoc is not", () => { + const root = mkdtempSync(join(tmpdir(), "scriptc-type-patches-js-")); + try { + const pkg = join(root, "node_modules/hoistkit"); + mkdirSync(pkg, { recursive: true }); + writeFileSync(join(pkg, "package.json"), '{ "name": "hoistkit", "version": "1.0.0" }'); + const source = [ + '"use strict";', + "exports.measure = size;", + 'exports.doc = "/** @typedef {object} Opts\\n * @property {number} width */";', + "", + "/**", + " * @returns {number}", + " */", + "function size(s) {", + " return s.length || undefined;", + "}", + "", + ].join("\n"); + writeFileSync(join(pkg, "index.js"), source); + writeFileSync( + join(root, "package.json"), + JSON.stringify({ + scriptc: { + typePatches: { + "hoistkit@^1.0.0": { + declarations: { "measure()": "| undefined", "Opts.width": "| undefined" }, + }, + }, + }, + }), + ); + configureTypePatches(join(root, "main.ts"), parse); + // The export, under another name, comes before the function it exports: + // the function's @returns tag widens. + expect(typePatchedText(join(pkg, "index.js"))).toBe( + source.replace(" * @returns {number}\n", " * @returns {number | undefined}\n"), + ); + // The string's text is a value, not a typedef: the patch finds nothing. + expect( + typePatchDiagnostics()! + .diagnostics.map((d) => d.message) + .join("\n"), + ).toContain("Opts.width"); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); diff --git a/packages/experimental/package.json b/packages/experimental/package.json index 583a2bc58..4636be674 100644 --- a/packages/experimental/package.json +++ b/packages/experimental/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/experimental", - "version": "0.4.4", + "version": "0.4.5", "description": "Experimental scriptc APIs, with Node.js implementations: lifetime regions for ProgramLifetime objects, intern tables shared between threads, and per-thread slots on shared objects", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-darwin-arm64/package.json b/packages/llvm-darwin-arm64/package.json index 76d5fdb72..d1ac54f3e 100644 --- a/packages/llvm-darwin-arm64/package.json +++ b/packages/llvm-darwin-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-darwin-arm64", - "version": "0.4.4", + "version": "0.4.5", "description": "Pinned LLVM code-generation helper for scriptc on macOS arm64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-darwin-x64/package.json b/packages/llvm-darwin-x64/package.json index fbb755f4a..b96a90cae 100644 --- a/packages/llvm-darwin-x64/package.json +++ b/packages/llvm-darwin-x64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-darwin-x64", - "version": "0.4.4", + "version": "0.4.5", "description": "Pinned LLVM code-generation helper for scriptc on macOS x64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-linux-arm64-gnu/package.json b/packages/llvm-linux-arm64-gnu/package.json index 81fdc7d5b..49796f522 100644 --- a/packages/llvm-linux-arm64-gnu/package.json +++ b/packages/llvm-linux-arm64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-linux-arm64-gnu", - "version": "0.4.4", + "version": "0.4.5", "description": "Pinned LLVM code-generation helper for scriptc on Linux arm64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-linux-arm64-musl/package.json b/packages/llvm-linux-arm64-musl/package.json index 8b3b2f185..3001949c6 100644 --- a/packages/llvm-linux-arm64-musl/package.json +++ b/packages/llvm-linux-arm64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-linux-arm64-musl", - "version": "0.4.4", + "version": "0.4.5", "description": "Statically linked LLVM code-generation helper for scriptc on Linux arm64 musl", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-linux-x64-gnu/package.json b/packages/llvm-linux-x64-gnu/package.json index 79b7eba3f..11b375ba4 100644 --- a/packages/llvm-linux-x64-gnu/package.json +++ b/packages/llvm-linux-x64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-linux-x64-gnu", - "version": "0.4.4", + "version": "0.4.5", "description": "Pinned LLVM code-generation helper for scriptc on Linux x64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-linux-x64-musl/package.json b/packages/llvm-linux-x64-musl/package.json index 5bffa5bef..ffe1ca7ae 100644 --- a/packages/llvm-linux-x64-musl/package.json +++ b/packages/llvm-linux-x64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-linux-x64-musl", - "version": "0.4.4", + "version": "0.4.5", "description": "Statically linked LLVM code-generation helper for scriptc on Linux x64 musl", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/llvm-win32-x64-msvc/package.json b/packages/llvm-win32-x64-msvc/package.json index 14c747a4a..77e6fe687 100644 --- a/packages/llvm-win32-x64-msvc/package.json +++ b/packages/llvm-win32-x64-msvc/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/llvm-win32-x64-msvc", - "version": "0.4.4", + "version": "0.4.5", "description": "Pinned LLVM code-generation helper for scriptc on Windows x64", "license": "Apache-2.0 WITH LLVM-exception", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-android-arm64/package.json b/packages/runtime-android-arm64/package.json index 1dfa1b177..7aea1da1e 100644 --- a/packages/runtime-android-arm64/package.json +++ b/packages/runtime-android-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-android-arm64", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc library runtime pack for Android arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-darwin-arm64/package.json b/packages/runtime-darwin-arm64/package.json index 32463357f..ae068a00a 100644 --- a/packages/runtime-darwin-arm64/package.json +++ b/packages/runtime-darwin-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-darwin-arm64", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for macOS arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-darwin-arm64/scripts/build.mjs b/packages/runtime-darwin-arm64/scripts/build.mjs index 7858c9782..3305d69de 100644 --- a/packages/runtime-darwin-arm64/scripts/build.mjs +++ b/packages/runtime-darwin-arm64/scripts/build.mjs @@ -366,7 +366,7 @@ async function build() { package: packageManifest.name, version: packageManifest.version, target: RUNTIME_PACK_MATRIX.target, - runtime_abi: { version: 44, marker: "scr_runtime_abi_v44" }, + runtime_abi: { version: 45, marker: "scr_runtime_abi_v45" }, compiler: { command: compiler, identity: compilerVersion, diff --git a/packages/runtime-darwin-x64/package.json b/packages/runtime-darwin-x64/package.json index c0577b04c..7dbdaf6b9 100644 --- a/packages/runtime-darwin-x64/package.json +++ b/packages/runtime-darwin-x64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-darwin-x64", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for macOS x64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-ios-arm64/package.json b/packages/runtime-ios-arm64/package.json index 9c2134280..8a868a606 100644 --- a/packages/runtime-ios-arm64/package.json +++ b/packages/runtime-ios-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-ios-arm64", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc library runtime pack for iOS arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-ios-simulator-arm64/package.json b/packages/runtime-ios-simulator-arm64/package.json index 6b7563a6a..011d41b39 100644 --- a/packages/runtime-ios-simulator-arm64/package.json +++ b/packages/runtime-ios-simulator-arm64/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-ios-simulator-arm64", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc library runtime pack for iOS simulator arm64", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-linux-arm64-gnu/package.json b/packages/runtime-linux-arm64-gnu/package.json index f0ae56929..92a3192e7 100644 --- a/packages/runtime-linux-arm64-gnu/package.json +++ b/packages/runtime-linux-arm64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-linux-arm64-gnu", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for Linux arm64 glibc", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-linux-arm64-musl/package.json b/packages/runtime-linux-arm64-musl/package.json index 79fa8e7dd..5e751d829 100644 --- a/packages/runtime-linux-arm64-musl/package.json +++ b/packages/runtime-linux-arm64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-linux-arm64-musl", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for Linux arm64 musl", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-linux-x64-gnu/package.json b/packages/runtime-linux-x64-gnu/package.json index 45af8d2b7..63c045385 100644 --- a/packages/runtime-linux-x64-gnu/package.json +++ b/packages/runtime-linux-x64-gnu/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-linux-x64-gnu", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for Linux x64 glibc", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-linux-x64-musl/package.json b/packages/runtime-linux-x64-musl/package.json index fe9d11bcd..a8602b395 100644 --- a/packages/runtime-linux-x64-musl/package.json +++ b/packages/runtime-linux-x64-musl/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-linux-x64-musl", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for Linux x64 musl", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-pack-common/scripts/build.mjs b/packages/runtime-pack-common/scripts/build.mjs index 9d9854689..be65002e4 100644 --- a/packages/runtime-pack-common/scripts/build.mjs +++ b/packages/runtime-pack-common/scripts/build.mjs @@ -469,7 +469,7 @@ async function build() { package: packageManifest.name, version: packageManifest.version, target: matrix.target, - runtime_abi: { version: 44, marker: "scr_runtime_abi_v44" }, + runtime_abi: { version: 45, marker: "scr_runtime_abi_v45" }, compiler: { command: compiler, identity: compilerVersion, diff --git a/packages/runtime-wasm32-wasi/package.json b/packages/runtime-wasm32-wasi/package.json index ac2f4a5e9..799863cf0 100644 --- a/packages/runtime-wasm32-wasi/package.json +++ b/packages/runtime-wasm32-wasi/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-wasm32-wasi", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for WASI Preview 1", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime-win32-x64-msvc/package.json b/packages/runtime-win32-x64-msvc/package.json index 9a5731da2..421f1f71f 100644 --- a/packages/runtime-win32-x64-msvc/package.json +++ b/packages/runtime-win32-x64-msvc/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime-win32-x64-msvc", - "version": "0.4.4", + "version": "0.4.5", "description": "Precompiled scriptc runtime pack for Windows x64 MSVC", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime/package.json b/packages/runtime/package.json index 1d58b00e5..90140ab15 100644 --- a/packages/runtime/package.json +++ b/packages/runtime/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/runtime", - "version": "0.4.4", + "version": "0.4.5", "description": "scriptc native runtime sources and vendored dependencies", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/runtime/src/scr_array.c b/packages/runtime/src/scr_array.c index 39998b8ff..9d4884390 100644 --- a/packages/runtime/src/scr_array.c +++ b/packages/runtime/src/scr_array.c @@ -269,11 +269,57 @@ static uint64_t scr_elem_retain_slot(const ScrArr *a, uint64_t slot) { return scr_elem_retain_with(a->elem, scr_arr_ops(a), slot); } +/* ToNumber and ToString of a foreign element. Both can run user code and + * pull in the dynamic tier, so they are reached through this table, which + * scr_arr_set_foreign installs: every foreign element is stored by it, and + * a program that never stores one links neither conversion. */ +typedef struct ScrArrForeignConversions { + double (*to_number)(const ScrDyn *); + ScrStr *(*to_string)(const ScrDyn *); +} ScrArrForeignConversions; + +static const ScrArrForeignConversions scr_arr_dyn_conversions = { + scr_dyn_number_coerce, + scr_dyn_string_coerce_js, +}; +static const ScrArrForeignConversions *scr_arr_conversions; + +static const ScrArrForeignConversions *scr_arr_foreign_conversions(void) { + return __atomic_load_n(&scr_arr_conversions, __ATOMIC_RELAXED); +} + +/* The array may hold SCR_ARR_FOREIGN elements (ScrArrExt.foreign). Such + * an array always has states. */ +static inline bool scr_arr_may_hold_foreign(const ScrArr *a) { + return SCR_AX(a)->foreign; +} + +/* `dst` takes slots of `src` (moved or retained): it may hold foreign + * elements too. Marking allocates the out-of-line storage, which can move + * the dense slots: mark before taking pointers into them. */ +static inline void scr_arr_inherit_foreign(ScrArr *dst, const ScrArr *src) { + if (scr_arr_may_hold_foreign(src)) scr_arr_ext(dst)->foreign = true; +} + static uint64_t scr_arr_retain_state_slot(const ScrArr *a, uint64_t slot, uint8_t state) { + if (state == SCR_ARR_FOREIGN) { + scr_dyn_retain(scr_slot_to_ptr(slot)); + return slot; + } return state == SCR_ARR_VALUE ? scr_elem_retain_slot(a, slot) : 0; } +/* Releases what a slot in `state` owns: a reference element, or the value + * of a foreign element. */ +static void scr_arr_release_state_slot(const ScrArr *a, uint64_t slot, uint8_t state) { + if (state == SCR_ARR_VALUE) { + if (scr_elem_is_ref(a->elem)) scr_elem_release(a, slot); + } else if (state == SCR_ARR_FOREIGN) { + scr_dyn_release(scr_slot_to_ptr(slot)); + } +} + /* ── storage and lifecycle ─────────────────────────────────────────────── */ typedef struct { @@ -981,6 +1027,7 @@ static inline bool scr_arr_slot_at(const ScrArr *a, size_t index, uint64_t *out) static void scr_arr_store_state_owned(ScrArr *a, size_t index, uint64_t slot, uint8_t state) { if (state != SCR_ARR_VALUE) scr_arr_cut(a, index); + if (state == SCR_ARR_FOREIGN) scr_arr_ext(a)->foreign = true; if (!scr_arr_has_states(a) && (state != SCR_ARR_VALUE || index != a->len)) scr_arr_add_states(a); scr_arr_grow_for_index(a, index); if (index < a->cap) { @@ -1034,6 +1081,7 @@ static void scr_arr_replace_state_owned(ScrArr *a, size_t index, uint64_t slot, uint8_t old_state; uint64_t old = 0; if (state != SCR_ARR_VALUE) scr_arr_cut(a, index); + if (state == SCR_ARR_FOREIGN) scr_arr_ext(a)->foreign = true; /* Without states, a value below the length replaces a value. */ if (!scr_arr_has_states(a) && (state != SCR_ARR_VALUE || index >= a->len)) { if (state == SCR_ARR_VALUE && index == a->len) { @@ -1044,7 +1092,7 @@ static void scr_arr_replace_state_owned(ScrArr *a, size_t index, uint64_t slot, } if (index < a->cap) { old_state = scr_arr_dense_state(a, index); - if (old_state == SCR_ARR_VALUE) old = scr_arr_dense_get(a, index); + if (old_state == SCR_ARR_VALUE || old_state == SCR_ARR_FOREIGN) old = scr_arr_dense_get(a, index); scr_arr_dense_put(a, index, slot); if (scr_arr_has_states(a)) *scr_arr_state_ptr(a, index) = state; } else { @@ -1058,7 +1106,7 @@ static void scr_arr_replace_state_owned(ScrArr *a, size_t index, uint64_t slot, SCR_AX(a)->sparse[pos].slot = slot; SCR_AX(a)->sparse[pos].state = state; } - if (old_state == SCR_ARR_VALUE && scr_elem_is_ref(a->elem)) scr_elem_release(a, old); + scr_arr_release_state_slot(a, old, old_state); } static void scr_arr_replace_owned(ScrArr *a, size_t index, uint64_t slot) { @@ -1300,6 +1348,7 @@ static void scr_arr_copy_dense(ScrArr *dst, size_t to, const ScrArr *src, size_t from, size_t count, bool reverse, bool materialize_holes) { if (count == 0) return; + scr_arr_inherit_foreign(dst, src); scr_arr_cut(dst, to); /* 8-byte slots: a constant slot size, and constant-width state ends. */ size_t width = scr_arr_slot_width(dst); @@ -1350,12 +1399,15 @@ static void scr_arr_copy_dense(ScrArr *dst, size_t to, const ScrArr *src, *scr_arr_state_in(dst_states, to + i) = values ? SCR_ARR_VALUE : *scr_arr_state_in(src_states, source); } } - if (!refs && (!materialize_holes || values)) return; + bool foreign = !values && scr_arr_may_hold_foreign(src); + if (!refs && !foreign && (!materialize_holes || values)) return; uint8_t elem = src->elem; const ScrArrOps *ops = scr_arr_ops(src); for (size_t i = to; i < to + count; i++) { if (*scr_arr_state_in(dst_states, i) == SCR_ARR_VALUE) { if (refs) dst->data[i] = scr_elem_retain_with(elem, ops, dst->data[i]); + } else if (*scr_arr_state_in(dst_states, i) == SCR_ARR_FOREIGN) { + scr_dyn_retain(scr_slot_to_ptr(dst->data[i])); } else if (materialize_holes && *scr_arr_state_in(dst_states, i) == SCR_ARR_HOLE) { scr_arr_dense_put(dst, i, 0); *scr_arr_state_in(dst_states, i) = SCR_ARR_UNDEFINED; @@ -1477,11 +1529,13 @@ void scr_arr_trace_v(void *a0, ScrTraceVisit visit, void *ctx) { } else { const uint8_t *states = scr_arr_states_end(a); for (size_t i = 0; i < end; i++) { - if (*scr_arr_state_in(states, i) == SCR_ARR_VALUE) visit(scr_slot_to_ptr(a->data[i]), ctx); + uint8_t state = *scr_arr_state_in(states, i); + if (state == SCR_ARR_VALUE || state == SCR_ARR_FOREIGN) visit(scr_slot_to_ptr(a->data[i]), ctx); } } for (size_t i = 0; i < SCR_AX(a)->sparse_len; i++) { - if (SCR_AX(a)->sparse[i].state == SCR_ARR_VALUE) { + uint8_t state = SCR_AX(a)->sparse[i].state; + if (state == SCR_ARR_VALUE || state == SCR_ARR_FOREIGN) { visit(scr_slot_to_ptr(SCR_AX(a)->sparse[i].slot), ctx); } } @@ -1590,11 +1644,37 @@ static inline __attribute__((always_inline)) void scr_arr_free(ScrArr *a) { scr_mem_free(a); } +/* Releases the values of foreign elements and leaves holes in their place, + * so the release below sees only typed elements. Arrays that hold foreign + * elements have states; a copy carries the mark of its source whatever + * range it takes, so an array with the mark and no states holds only + * values in its dense slots. */ +static void scr_arr_release_foreign(ScrArr *a) { + uint8_t *states = scr_arr_has_states(a) ? scr_arr_states_end(a) : NULL; + for (size_t i = 0, end = states ? (a->len < a->cap ? a->len : a->cap) : 0; i < end; i++) { + if (*scr_arr_state_in(states, i) != SCR_ARR_FOREIGN) continue; + uint64_t old = a->data[i]; + *scr_arr_state_in(states, i) = SCR_ARR_HOLE; + a->data[i] = 0; + scr_dyn_release(scr_slot_to_ptr(old)); + } + ScrArrExt *x = scr_arr_ext_of(a); + for (size_t i = 0; i < x->sparse_len; i++) { + if (x->sparse[i].state != SCR_ARR_FOREIGN) continue; + uint64_t old = x->sparse[i].slot; + x->sparse[i].state = SCR_ARR_HOLE; + x->sparse[i].slot = 0; + scr_dyn_release(scr_slot_to_ptr(old)); + } + x->foreign = false; +} + static void scr_arr_destroy(void *object) { ScrArr *a = object; if (scr_arr_ext_of(a)) { scr_dyn_release(scr_arr_ext_of(a)->metadata); scr_arr_ext_of(a)->metadata = NULL; + if (scr_arr_ext_of(a)->foreign) scr_arr_release_foreign(a); } if (scr_elem_is_ref(a->elem)) { size_t end = a->len < a->cap ? a->len : a->cap; @@ -1898,14 +1978,14 @@ void scr_arr_set_len(ScrArr *a, double length) { } return; } - if (scr_elem_is_ref(a->elem)) { + if (scr_elem_is_ref(a->elem) || scr_arr_may_hold_foreign(a)) { for (size_t i = dense_stop; i < dense_end; i++) { if (*scr_arr_state_ptr(a, i) != SCR_ARR_HOLE) { uint64_t old = a->data[i]; uint8_t state = *scr_arr_state_ptr(a, i); *scr_arr_state_ptr(a, i) = 0; a->data[i] = 0; - if (state == SCR_ARR_VALUE) scr_elem_release(a, old); + scr_arr_release_state_slot(a, old, state); } } if (scr_arr_ext_of(a)) { @@ -1915,7 +1995,7 @@ void scr_arr_set_len(ScrArr *a, double length) { uint64_t old = scr_arr_ext_of(a)->sparse[i].slot; uint8_t state = scr_arr_ext_of(a)->sparse[i].state; scr_arr_ext_of(a)->sparse_len = i; - if (state == SCR_ARR_VALUE) scr_elem_release(a, old); + scr_arr_release_state_slot(a, old, state); } } } else { @@ -1949,7 +2029,7 @@ void scr_arr_copy_index(ScrArr *dst, size_t dst_index, const ScrArr *src, uint64_t slot; uint8_t state = scr_arr_state_at(src, src_index, &slot); if (state == SCR_ARR_HOLE) return; - if (state == SCR_ARR_VALUE) slot = scr_elem_retain_slot(src, slot); + slot = scr_arr_retain_state_slot(src, slot, state); scr_arr_replace_state_owned(dst, dst_index, slot, state); } @@ -2056,7 +2136,7 @@ ScrArr *scr_arr_copy_within(ScrArr *a, double target, double start, double end) if (state == SCR_ARR_HOLE) { scr_arr_delete(a, (double)(to + offset)); } else { - if (state == SCR_ARR_VALUE) slot = scr_elem_retain_slot(a, slot); + slot = scr_arr_retain_state_slot(a, slot, state); /* Each replacement publishes its edge before releasing the previous * owner; cycle collection can run between successive positions. */ scr_arr_replace_state_owned(a, to + offset, slot, state); @@ -2074,7 +2154,7 @@ static ScrArr *scr_arr_fill_slot(ScrArr *a, uint64_t slot, uint8_t state, bool refs = scr_elem_is_ref(a->elem); if (scr_arr_dense_ok(a, until, until - from)) scr_arr_grow_dense(a, until); if (state != SCR_ARR_VALUE) scr_arr_need_states(a); - if (!refs && until <= a->cap) { + if (!refs && until <= a->cap && !scr_arr_may_hold_foreign(a)) { if (state != SCR_ARR_VALUE) scr_arr_cut(a, from); if (a->elem == SCR_ELEM_BOOL) memset(scr_arr_dense_at(a, from), (int)slot, until - from); else if (a->elem == SCR_ELEM_CREF) for (size_t i = from; i < until; i++) scr_arr_dense_put(a, i, slot); @@ -2114,7 +2194,63 @@ ScrArr *scr_arr_fill_undefined(ScrArr *a, double start, double end) { * The JS fold exactly (ECMA Math.max/min applied to the elements): any * NaN poisons the result, +0 beats -0 for max (the reverse for min), and * the empty array yields the zero-argument constants. Borrows the array. */ +/* ToNumber of every element in order, as Math.max/min/hypot apply it to + * their spread arguments before comparing; for arrays that may hold + * foreign elements, whose conversion can call user code. As in the spread, + * the elements are taken first: the foreign values are retained before + * any conversion runs, so a conversion that changes the array changes + * neither the count (*count) nor the values. NULL with the exception + * pending when a conversion throws. +1 malloc'd buffer. */ +static double *scr_arr_coerced_numbers(const ScrArr *a, size_t *count) { + size_t n = a->len; + /* A sparse array's length can exceed what the buffers can count. */ + if (n > SIZE_MAX / sizeof(double)) scr_arr_oom(); + double *out = malloc((n ? n : 1) * sizeof(*out)); + ScrDyn **held = calloc(n ? n : 1, sizeof(*held)); + if (!out || !held) scr_arr_oom(); + for (size_t i = 0; i < n; i++) { + uint64_t slot; + uint8_t state = scr_arr_f64_state_at(a, i, &slot); + if (state == SCR_ARR_FOREIGN) held[i] = scr_dyn_retain(scr_slot_to_ptr(slot)); + out[i] = state == SCR_ARR_VALUE ? scr_slot_to_f64(slot) : NAN; + } + bool failed = false; + for (size_t i = 0; i < n; i++) { + if (!held[i]) continue; + if (!failed) { + out[i] = scr_arr_foreign_conversions()->to_number(held[i]); + failed = scr_exc_pending(); + } + scr_dyn_release(held[i]); + } + free(held); + if (failed) { + free(out); + return NULL; + } + *count = n; + return out; +} + +static double scr_math_extreme_foreign(ScrArr *a, bool max) { + size_t n; + double *values = scr_arr_coerced_numbers(a, &n); + if (!values) return NAN; + double best = max ? -INFINITY : INFINITY; + bool nan = false; + for (size_t i = 0; i < n; i++) { + double v = values[i]; + if (isnan(v)) nan = true; + else if (max ? (v > best || (v == 0.0 && best == 0.0 && !signbit(v))) + : (v < best || (v == 0.0 && best == 0.0 && signbit(v)))) + best = v; + } + free(values); + return nan ? NAN : best; +} + double scr_math_max_arr(ScrArr *a) { + if (scr_arr_may_hold_foreign(a)) return scr_math_extreme_foreign(a, true); double best = -INFINITY; for (size_t i = 0; i < a->len; i++) { uint64_t slot; @@ -2127,6 +2263,7 @@ double scr_math_max_arr(ScrArr *a) { } double scr_math_min_arr(ScrArr *a) { + if (scr_arr_may_hold_foreign(a)) return scr_math_extreme_foreign(a, false); double best = INFINITY; for (size_t i = 0; i < a->len; i++) { uint64_t slot; @@ -2143,30 +2280,43 @@ double scr_math_min_arr(ScrArr *a) { double scr_math_hypot_arr(ScrArr *a) { double scale = 0.0; bool has_nan = false; - for (size_t i = 0; i < a->len; i++) { + double *coerced = NULL; + size_t n = a->len; + if (scr_arr_may_hold_foreign(a) && !(coerced = scr_arr_coerced_numbers(a, &n))) return NAN; + for (size_t i = 0; i < n; i++) { uint64_t slot; - if (scr_arr_f64_state_at(a, i, &slot) != SCR_ARR_VALUE) { + double v; + if (coerced) { + v = fabs(coerced[i]); + } else if (scr_arr_f64_state_at(a, i, &slot) != SCR_ARR_VALUE) { has_nan = true; continue; + } else { + v = fabs(scr_slot_to_f64(slot)); + } + if (isinf(v)) { + free(coerced); + return INFINITY; } - double v = fabs(scr_slot_to_f64(slot)); - if (isinf(v)) return INFINITY; if (isnan(v)) has_nan = true; else if (v > scale) scale = v; } - if (has_nan) return NAN; - if (scale == 0.0) return 0.0; + if (has_nan || scale == 0.0) { + free(coerced); + return has_nan ? NAN : 0.0; + } double sum = 0.0; double correction = 0.0; - for (size_t i = 0; i < a->len; i++) { - uint64_t slot; - (void)scr_arr_f64_state_at(a, i, &slot); - double v = scr_slot_to_f64(slot) / scale; + for (size_t i = 0; i < n; i++) { + uint64_t slot = 0; + if (!coerced) (void)scr_arr_f64_state_at(a, i, &slot); + double v = (coerced ? coerced[i] : scr_slot_to_f64(slot)) / scale; double term = v * v - correction; double next = sum + term; correction = (next - sum) - term; sum = next; } + free(coerced); return scale * sqrt(sum); } @@ -2211,6 +2361,13 @@ static uint64_t scr_arr_require_index(const ScrArr *a, size_t idx, double i) { scr_trap_fmt("scriptc: RangeError: array index %s is undefined (length %zu)\n", buf, a->len); } + if (state == SCR_ARR_FOREIGN) { + char buf[32]; + scr_f64_to_str(i, buf); + scr_trap_fmt("scriptc: RangeError: array index %s holds a value outside its element type " + "(length %zu)\n", + buf, a->len); + } return slot; } @@ -2218,16 +2375,20 @@ double scr_arr_get_f64(ScrArr *a, double i) { return scr_slot_to_f64(scr_arr_require_slot(a, i)); } -static bool scr_arr_read_value(const ScrArr *a, double i, uint64_t *slot) { +/* The state of a[i] for any numeric key (a hole when missing), with its + * slot. */ +static uint8_t scr_arr_read_state(const ScrArr *a, double i, uint64_t *slot) { size_t idx; uint8_t state; if (scr_arr_valid_index(i, &idx)) { - if (idx >= a->len) return false; - state = scr_arr_state_at(a, idx, slot); - } else if (!scr_arr_prop_get_state(a, i, slot, &state)) { - return false; + if (idx >= a->len) return SCR_ARR_HOLE; + return scr_arr_state_at(a, idx, slot); } - return state == SCR_ARR_VALUE; + return scr_arr_prop_get_state(a, i, slot, &state) ? state : SCR_ARR_HOLE; +} + +static bool scr_arr_read_value(const ScrArr *a, double i, uint64_t *slot) { + return scr_arr_read_state(a, i, slot) == SCR_ARR_VALUE; } void *scr_arr_peek_ref(const ScrArr *a, double i) { @@ -2257,6 +2418,13 @@ double scr_arr_get_number(const ScrArr *a, double i) { } else if (!scr_arr_prop_get_state(a, i, &slot, &state)) { return NAN; } + if (state == SCR_ARR_FOREIGN) { + /* The conversion can call user code that removes the element. */ + ScrDyn *value = scr_dyn_retain(scr_slot_to_ptr(slot)); + double number = scr_arr_foreign_conversions()->to_number(value); + scr_dyn_release(value); + return number; + } return state == SCR_ARR_VALUE ? scr_slot_to_f64(slot) : NAN; } @@ -2267,14 +2435,26 @@ bool scr_arr_index_eq(const ScrArr *a, double i, const ScrArr *b, double j) { if (i >= 0 && i < (double)a->cap && i == trunc(i) && j >= 0 && j < (double)b->cap && j == trunc(j)) { size_t ai = (size_t)i, bi = (size_t)j; - bool ap = ai < a->len && scr_arr_dense_state(a, ai) == SCR_ARR_VALUE; - bool bp = bi < b->len && scr_arr_dense_state(b, bi) == SCR_ARR_VALUE; + uint8_t as = ai < a->len ? scr_arr_dense_state(a, ai) : SCR_ARR_HOLE; + uint8_t bs = bi < b->len ? scr_arr_dense_state(b, bi) : SCR_ARR_HOLE; + if (as == SCR_ARR_FOREIGN || bs == SCR_ARR_FOREIGN) { + /* A foreign element is never a value of the element type, undefined + * or a hole. */ + return as == bs && scr_dyn_strict_eq(scr_slot_to_ptr(scr_arr_dense_get(a, ai)), + scr_slot_to_ptr(scr_arr_dense_get(b, bi))); + } + bool ap = as == SCR_ARR_VALUE; + bool bp = bs == SCR_ARR_VALUE; if (!ap || !bp) return ap == bp; av = scr_arr_dense_get(a, ai); bv = scr_arr_dense_get(b, bi); } else { - bool ap = scr_arr_read_value(a, i, &av); - bool bp = scr_arr_read_value(b, j, &bv); + uint8_t as = scr_arr_read_state(a, i, &av); + uint8_t bs = scr_arr_read_state(b, j, &bv); + if (as == SCR_ARR_FOREIGN || bs == SCR_ARR_FOREIGN) + return as == bs && scr_dyn_strict_eq(scr_slot_to_ptr(av), scr_slot_to_ptr(bv)); + bool ap = as == SCR_ARR_VALUE; + bool bp = bs == SCR_ARR_VALUE; if (!ap || !bp) return ap == bp; } if (a->elem == SCR_ELEM_F64) return scr_slot_to_f64(av) == scr_slot_to_f64(bv); @@ -2349,6 +2529,44 @@ void scr_arr_set_undefined(ScrArr *a, double i) { scr_arr_replace_state_owned(a, idx, 0, SCR_ARR_UNDEFINED); } +bool scr_arr_set_foreign(ScrArr *a, double i, ScrDyn *v) { + size_t idx; + if (!scr_arr_valid_index(i, &idx)) return false; + if (v->kind == SCR_DYN_UNDEF) { + scr_arr_set_undefined(a, i); + return true; + } + /* A foreign element keeps a pointer in its slot: int32 slots widen to + * doubles first, and 1-byte and compact slots cannot hold one. */ + if (a->elem == SCR_ELEM_I32) scr_arr_widen_i32(a); + if (scr_arr_slot_width(a) != sizeof(uint64_t)) return false; + __atomic_store_n(&scr_arr_conversions, &scr_arr_dyn_conversions, __ATOMIC_RELAXED); + scr_dyn_retain(v); + scr_arr_need_states(a); + if (idx >= a->len) a->len = idx + 1; + scr_arr_replace_state_owned(a, idx, scr_slot_from_ptr(v), SCR_ARR_FOREIGN); + return true; +} + +ScrDyn *scr_arr_get_foreign(const ScrArr *a, double i) { + size_t idx; + uint64_t slot; + if (!scr_arr_valid_index(i, &idx) || idx >= a->len) return NULL; + if (scr_arr_state_at(a, idx, &slot) != SCR_ARR_FOREIGN) return NULL; + return scr_dyn_retain(scr_slot_to_ptr(slot)); +} + +void scr_arr_foreign_throw(const ScrArr *a, double i, const ScrStr *want) { + ScrDyn *value = scr_arr_get_foreign(a, i); + scr_dyn_check_fail(NULL, want->data, value ? value : scr_dyn_undefined()); + scr_dyn_release(value); +} + +void scr_dyn_foreign_throw(ScrDyn *value, const char *want) { + scr_dyn_check_fail(NULL, want, value); + scr_dyn_release(value); +} + bool scr_arr_delete(ScrArr *a, double i) { size_t idx; if (!scr_arr_valid_index(i, &idx)) return scr_arr_prop_delete(a, i); @@ -2356,9 +2574,7 @@ bool scr_arr_delete(ScrArr *a, double i) { uint64_t old; uint8_t state; if (!scr_arr_take_state(a, idx, &old, &state)) return false; - if (state == SCR_ARR_VALUE && scr_elem_is_ref(a->elem)) { - scr_elem_release(a, old); - } + scr_arr_release_state_slot(a, old, state); return true; } @@ -2429,8 +2645,8 @@ double scr_arr_push_spread(ScrArr *a, const ScrArr *src) { uint64_t slot; uint8_t state = scr_arr_state_at_storage(&old, i, &slot); if (state != SCR_ARR_HOLE) scr_arr_store_state_owned(a, i, slot, state); - if (state == SCR_ARR_VALUE) { - scr_arr_store_owned(a, old_len + i, scr_elem_retain_slot(a, slot)); + if (state == SCR_ARR_VALUE || state == SCR_ARR_FOREIGN) { + scr_arr_store_state_owned(a, old_len + i, scr_arr_retain_state_slot(a, slot, state), state); } else { scr_arr_store_state_owned(a, old_len + i, 0, SCR_ARR_UNDEFINED); } @@ -2445,8 +2661,8 @@ double scr_arr_push_spread(ScrArr *a, const ScrArr *src) { for (size_t i = 0; i < add; i++) { uint64_t slot; uint8_t state = scr_arr_state_at(src, i, &slot); - if (state == SCR_ARR_VALUE) { - scr_arr_store_owned(a, old_len + i, scr_elem_retain_slot(src, slot)); + if (state == SCR_ARR_VALUE || state == SCR_ARR_FOREIGN) { + scr_arr_store_state_owned(a, old_len + i, scr_arr_retain_state_slot(src, slot, state), state); } else { scr_arr_store_state_owned(a, old_len + i, 0, SCR_ARR_UNDEFINED); } @@ -2474,11 +2690,8 @@ double scr_arr_concat_copy(ScrArr *a, const ScrArr *src) { uint64_t slot; uint8_t state = scr_arr_state_at_storage(&old, i, &slot); if (state != SCR_ARR_HOLE) scr_arr_store_state_owned(a, i, slot, state); - if (state == SCR_ARR_VALUE) { - scr_arr_store_owned(a, old_len + i, scr_elem_retain_slot(a, slot)); - } else if (state == SCR_ARR_UNDEFINED) { - scr_arr_store_state_owned(a, old_len + i, 0, state); - } + if (state != SCR_ARR_HOLE) + scr_arr_store_state_owned(a, old_len + i, scr_arr_retain_state_slot(a, slot, state), state); } a->len = old_len + add; scr_arr_free_storage(&old); @@ -2659,8 +2872,9 @@ double scr_arr_unshift_spread(ScrArr *a, const ScrArr *src) { uint8_t state = src == a ? scr_arr_state_at_storage(&old, i, &slot) : scr_arr_state_at(src, i, &slot); - if (state == SCR_ARR_VALUE) { - scr_arr_store_owned(a, i, scr_arr_retain_state_slot(src == a ? a : src, slot, state)); + if (state == SCR_ARR_VALUE || state == SCR_ARR_FOREIGN) { + scr_arr_store_state_owned(a, i, scr_arr_retain_state_slot(src == a ? a : src, slot, state), + state); } else { scr_arr_store_state_owned(a, i, 0, SCR_ARR_UNDEFINED); } @@ -2769,7 +2983,8 @@ uint8_t scr_arr_shift_state(ScrArr *a, uint64_t *slot_out) { size_t old_len = a->len; if (scr_arr_is_dense(a)) { uint8_t state = scr_arr_dense_state(a, 0); - if (slot_out && state == SCR_ARR_VALUE) *slot_out = scr_arr_dense_get(a, 0); + if (slot_out && (state == SCR_ARR_VALUE || state == SCR_ARR_FOREIGN)) + *slot_out = scr_arr_dense_get(a, 0); /* The first slot becomes head room; the tail stays where it is. */ if (scr_arr_drop_front(a, 1)) return state; /* The prefix slides down with the tail. */ @@ -2842,6 +3057,7 @@ ScrArr *scr_arr_splice(ScrArr *a, double start, double deleteCount) { size_t old_len = a->len; if (scr_arr_is_dense(a)) { if (n) { + scr_arr_inherit_foreign(out, a); /* The removed elements keep their states unless all are values. */ if (!scr_arr_dense_run_values(a, from, n)) scr_arr_add_states(out); memcpy(out->data, scr_arr_dense_at(a, from), n * scr_arr_slot_width(a)); @@ -2927,6 +3143,7 @@ ScrArr *scr_arr_splice_insert(ScrArr *a, double start, double deleteCount, ? scr_arr_new_ops(scr_arr_ops(a), n) : scr_arr_new(a->elem, n); if (n) { + scr_arr_inherit_foreign(removed, a); if (!scr_arr_dense_run_values(a, from, n)) scr_arr_add_states(removed); memcpy(removed->data, scr_arr_dense_at(a, from), n * scr_arr_slot_width(a)); if (scr_arr_has_states(removed)) @@ -2981,22 +3198,30 @@ void scr_arr_splice_drop(ScrArr *a, double start, double deleteCount, size_t cou } /* Removed slots, kept until the receiver is rewritten. */ uint64_t inline_removed[8]; + uint8_t inline_states[8]; uint64_t *removed = inline_removed; + uint8_t *removed_states = inline_states; size_t kept = 0; - if (refs && n > 0) { + if ((refs || scr_arr_may_hold_foreign(a)) && n > 0) { if (n > 8) { removed = scr_mem_alloc(n * sizeof(uint64_t)); - if (!removed) scr_arr_oom(); + removed_states = scr_mem_alloc(n); + if (!removed || !removed_states) scr_arr_oom(); } for (size_t i = from; i < from + n; i++) { - if (scr_arr_dense_state(a, i) == SCR_ARR_VALUE) removed[kept++] = a->data[i]; + uint8_t state = scr_arr_dense_state(a, i); + if ((refs && state == SCR_ARR_VALUE) || state == SCR_ARR_FOREIGN) { + removed_states[kept] = state; + removed[kept++] = a->data[i]; + } } } size_t old_len = a->len; if (from == 0 && count == 0 && scr_arr_drop_front(a, n)) { /* A removed prefix becomes head room. */ - for (size_t i = 0; i < kept; i++) scr_elem_release(a, removed[i]); + for (size_t i = 0; i < kept; i++) scr_arr_release_state_slot(a, removed[i], removed_states[i]); if (removed != inline_removed) scr_mem_free(removed); + if (removed_states != inline_states) scr_mem_free(removed_states); return; } scr_arr_grow_dense(a, next_len); @@ -3011,8 +3236,9 @@ void scr_arr_splice_drop(ScrArr *a, double start, double deleteCount, size_t cou /* Values inserted into a hole-free array keep it hole-free. */ if (packed) a->filled = (uint32_t)next_len; else scr_arr_refill(a); - for (size_t i = 0; i < kept; i++) scr_elem_release(a, removed[i]); + for (size_t i = 0; i < kept; i++) scr_arr_release_state_slot(a, removed[i], removed_states[i]); if (removed != inline_removed) scr_mem_free(removed); + if (removed_states != inline_states) scr_mem_free(removed_states); } /* FlattenIntoArray for static arrays: a dense copy with depth zero, or one @@ -3236,7 +3462,10 @@ typedef struct { typedef struct { ScrElemKind kind; const ScrNumberSortKey *numbers; - ScrStr *const *texts; /* reference elements: each element's String() text */ + /* When set, values are indices into these string keys: each element's + * String() text (reference elements such as Dates, and the elements of an + * array that may hold foreign elements). */ + ScrStr *const *texts; } ScrPrimitiveOrder; static int scr_arr_primitive_compare(uint64_t left, uint64_t right, @@ -3329,6 +3558,110 @@ static void scr_arr_primitive_order(uint64_t *values, size_t count, if (runs != local_runs) free(runs); } +/* The default ordering of an array that may hold foreign elements: present + * elements sort by their string conversions, then present undefined + * values, then holes (toSorted reads them as undefined). The conversions + * may call user code, so the elements are taken first, each retained, as + * the sort's element list is; a throw leaves the array unchanged. */ +static ScrArr *scr_arr_sort_foreign(ScrArr *a, bool copy) { + size_t length = a->len; + size_t dense = a->len < a->cap ? a->len : a->cap; + size_t cap = dense + SCR_AX(a)->sparse_len; + if (cap > SIZE_MAX / sizeof(uint64_t)) scr_arr_oom(); + uint64_t *slots = cap ? malloc(cap * sizeof(*slots)) : NULL; + uint8_t *states = cap ? malloc(cap) : NULL; + ScrStr **keys = cap ? malloc(cap * sizeof(*keys)) : NULL; + uint64_t *order_of = cap ? malloc(cap * sizeof(*order_of)) : NULL; + if (cap && (!slots || !states || !keys || !order_of)) scr_arr_oom(); + size_t count = 0, undefined_count = 0; + for (size_t at = 0; at < cap; at++) { + uint64_t slot; + uint8_t state; + if (at < dense) { + state = scr_arr_dense_state(a, at); + slot = scr_arr_dense_get(a, at); + } else { + state = SCR_AX(a)->sparse[at - dense].state; + slot = SCR_AX(a)->sparse[at - dense].slot; + } + if (state == SCR_ARR_UNDEFINED) undefined_count++; + if (state != SCR_ARR_VALUE && state != SCR_ARR_FOREIGN) continue; + slots[count] = scr_arr_retain_state_slot(a, slot, state); + states[count] = state; + count++; + } + bool failed = false; + size_t keyed = 0; + for (; keyed < count; keyed++) { + uint64_t slot = slots[keyed]; + ScrStr *key; + if (states[keyed] == SCR_ARR_FOREIGN) { + key = scr_arr_foreign_conversions()->to_string(scr_slot_to_ptr(slot)); + } else if (a->elem == SCR_ELEM_STR) { + key = scr_str_retain(scr_slot_to_ptr(slot)); + } else { + char text[32]; + size_t n = scr_f64_to_str(scr_slot_to_f64(slot), text); + key = scr_str_new(text, n); + } + if (!key) { + failed = true; + break; + } + keys[keyed] = key; + order_of[keyed] = keyed; + } + ScrArr *out = a; + if (!failed) { + ScrPrimitiveOrder order = {a->elem, NULL, keys}; + scr_arr_primitive_order(order_of, count, &order); + if (copy) { + /* The values, then present undefined for every other index. */ + out = scr_arr_has_ops(a) ? scr_arr_new_ops(scr_arr_ops(a), length) + : scr_arr_new(a->elem, length); + for (size_t i = 0; i < length; i++) { + if (i < count) scr_arr_store_state_owned(out, i, slots[order_of[i]], states[order_of[i]]); + else scr_arr_store_state_owned(out, i, 0, SCR_ARR_UNDEFINED); + out->len = (uint32_t)(i + 1); + } + } else { + /* The values, then the present undefined values; holes after them + * up to the sorted length. Elements a conversion added at or past + * that length stay, and the array keeps the longer of its current + * length and the sorted count, as in Node. The taken elements own + * their references; release the ones the array holds now below the + * sorted length, which a conversion may have changed. */ + size_t now = a->len; + size_t sorted = count + undefined_count; + ScrArrStorage old = scr_arr_take_storage(a); + a->len = (uint32_t)(now > sorted ? now : sorted); + for (size_t i = 0; i < count; i++) + scr_arr_store_state_owned(a, i, slots[order_of[i]], states[order_of[i]]); + for (size_t i = count; i < sorted; i++) scr_arr_store_state_owned(a, i, 0, SCR_ARR_UNDEFINED); + size_t old_dense = now < old.cap ? now : old.cap; + for (size_t i = 0; i < old_dense + old.sparse_len; i++) { + size_t index = i < old_dense ? i : old.sparse[i - old_dense].index; + uint64_t slot = 0; + uint8_t state = scr_arr_state_at_storage(&old, index, &slot); + if (state == SCR_ARR_HOLE) continue; + if (index < length) scr_arr_release_state_slot(a, slot, state); + else scr_arr_store_state_owned(a, index, slot, state); + } + scr_arr_free_storage(&old); + } + scr_arr_refill(out); + } else { + for (size_t i = 0; i < count; i++) scr_arr_release_state_slot(a, slots[i], states[i]); + } + for (size_t i = 0; i < keyed; i++) scr_str_release(keys[i]); + free(slots); + free(states); + free(keys); + free(order_of); + if (failed) return copy ? scr_arr_new(a->elem, 0) : scr_arr_retain(a); + return copy ? out : scr_arr_retain(a); +} + static ScrArr *scr_arr_sort_default(ScrArr *a, bool copy, ScrStr *(*text)(void *)); static ScrArr *scr_arr_new_like(const ScrArr *a, size_t cap) { @@ -3338,6 +3671,7 @@ static ScrArr *scr_arr_new_like(const ScrArr *a, size_t cap) { ScrArr *scr_arr_sort_primitive(ScrArr *a, bool copy) { if (a->elem != SCR_ELEM_STR && a->elem != SCR_ELEM_F64 && a->elem != SCR_ELEM_BOOL) scr_trap("scriptc: invalid primitive array ordering\n"); + if (a->len > 0 && scr_arr_may_hold_foreign(a)) return scr_arr_sort_foreign(a, copy); return scr_arr_sort_default(a, copy, NULL); } @@ -5190,7 +5524,8 @@ bool scr_arr_sort_dense(ScrArr *a, ScrClosure *f, uint32_t arity, bool owned, in for (size_t i = 0; i < n; i++) scr_elem_release_with(elem, ops, src[i]); scr_array_frozen_error(0, a->len, 0, 0, 0); } else if (a->len >= n && a->cap >= n) { - if (!retained) { + /* The comparator may have stored foreign elements through a view. */ + if (!retained && !scr_arr_may_hold_foreign(a)) { memcpy(a->data, src, n * sizeof(*src)); if (scr_arr_has_states(a)) memset(scr_arr_state_run(a, 0, n), SCR_ARR_VALUE, n); } else { @@ -5198,11 +5533,11 @@ bool scr_arr_sort_dense(ScrArr *a, ScrClosure *f, uint32_t arity, bool owned, in * release can collect cycles. The buffer still holds the slots not * yet written. */ for (size_t i = 0; i < n; i++) { - bool held = scr_arr_dense_state(a, i) == SCR_ARR_VALUE; - uint64_t old = held ? a->data[i] : 0; + uint8_t old_state = scr_arr_dense_state(a, i); + uint64_t old = old_state == SCR_ARR_VALUE || old_state == SCR_ARR_FOREIGN ? a->data[i] : 0; a->data[i] = src[i]; if (scr_arr_has_states(a)) *scr_arr_state_ptr(a, i) = SCR_ARR_VALUE; - if (held) scr_elem_release(a, old); + scr_arr_release_state_slot(a, old, old_state); } } } else { @@ -5270,10 +5605,28 @@ ScrStr *scr_arr_join(ScrArr *a, ScrStr *sep) { * across its seam; the finished string then needs fusing. */ bool lone = scr_wtf8_any(); bool lead_low = lone && a->len > 1 && scr_wtf8_low_at(sep->data); - for (size_t i = 0; i < a->len; i++) { + /* The length is read once: a conversion that shortens the array leaves + * the indices past its new end empty. */ + size_t length = a->len; + for (size_t i = 0; i < length; i++) { if (i > 0) scr_join_append(&out, sep->data, sep->len); + if (i >= a->len) continue; uint64_t slot; uint8_t state = scr_arr_state_at(a, i, &slot); + if (state == SCR_ARR_FOREIGN) { + /* join converts every element but undefined and null with ToString, + * which can call user code that removes the element; a throw ends + * the join with the exception pending. */ + ScrDyn *value = scr_slot_to_ptr(slot); + if (value->kind == SCR_DYN_UNDEF || value->kind == SCR_DYN_NULL) continue; + scr_dyn_retain(value); + ScrStr *text = scr_arr_foreign_conversions()->to_string(value); + scr_dyn_release(value); + if (!text) break; + scr_join_append(&out, text->data, text->len); + scr_str_release(text); + continue; + } if (state != SCR_ARR_VALUE) continue; switch (a->elem) { case SCR_ELEM_F64: { diff --git a/packages/runtime/src/scr_array.h b/packages/runtime/src/scr_array.h index 48fa4f26f..fab85fff1 100644 --- a/packages/runtime/src/scr_array.h +++ b/packages/runtime/src/scr_array.h @@ -60,6 +60,10 @@ typedef enum { SCR_ARR_HOLE = 0, SCR_ARR_VALUE = 1, SCR_ARR_UNDEFINED = 2, + /* A present element outside the element type, stored through a dynamic + * view: the slot holds a retained ScrDyn * (scr_arr_set_foreign). Arrays + * that hold one mark their out-of-line storage (ScrArrExt.foreign). */ + SCR_ARR_FOREIGN = 3, } ScrArrState; /* The RC entry points of an SCR_ELEM_REF array's element type: one constant @@ -171,6 +175,9 @@ typedef struct ScrArrExt { /* The element ops of an array whose ops have no registry id * (SCR_ARR_OPS_OVERFLOW). */ const ScrArrOps *ops; + /* Some slot may hold SCR_ARR_FOREIGN; release and copies walk the states + * for their payloads only then. */ + bool foreign; } ScrArrExt; /* The LLVM backend mirrors ScrArr as %ScrArr: its inline retain/release read * `cyc` (field 4, i8) to decide mark-live / candidate buffering exactly like @@ -595,6 +602,20 @@ void scr_arr_set_f64(ScrArr *a, double i, double v); void scr_arr_set_bool(ScrArr *a, double i, bool v); void scr_arr_set_ref(ScrArr *a, double i, void *v); void scr_arr_set_undefined(ScrArr *a, double i); +/* Stores `v` (borrowed, retained) at a canonical index as a value outside + * the element type (SCR_ARR_FOREIGN), or as a present undefined when it is + * undefined, extending the length like the typed setters. False, with + * nothing stored, for an index that is not canonical and for a foreign + * value in a boolean array, whose 1-byte dense slots cannot hold it. */ +bool scr_arr_set_foreign(ScrArr *a, double i, struct ScrDyn *v); +/* The value of a foreign element (+1), or NULL when a[i] is not one. */ +struct ScrDyn *scr_arr_get_foreign(const ScrArr *a, double i); +/* Throws the checked conversion's TypeError for the foreign element a[i] + * read where a value of type `want` is expected. */ +void scr_arr_foreign_throw(const ScrArr *a, double i, const ScrStr *want); +/* The same TypeError for a foreign element's value removed from an array + * (pop/shift into a typed result); takes and releases `value`. */ +void scr_dyn_foreign_throw(struct ScrDyn *value, const char *want); bool scr_arr_delete(ScrArr *a, double i); /* slice(start?, end?): a fresh +1 shallow copy of the index range — diff --git a/packages/runtime/src/scr_async.c b/packages/runtime/src/scr_async.c index dfc4e9825..0f18756e1 100644 --- a/packages/runtime/src/scr_async.c +++ b/packages/runtime/src/scr_async.c @@ -4934,7 +4934,7 @@ void scr_loop_hosted_run(uint32_t kind, uint32_t token) { ScrClosure *cb = t->cb; ScrFsRenameFn error_cb = t->error_cb; void (*raw)(void) = t->raw; - free(t); + scr_mem_free(t); if (cb) { if (error_cb) error_cb(cb, NULL); else ((void (*)(ScrClosure *))cb->fn)(cb); diff --git a/packages/runtime/src/scr_bytes.c b/packages/runtime/src/scr_bytes.c index df64b2ab2..5fb8a58d4 100644 --- a/packages/runtime/src/scr_bytes.c +++ b/packages/runtime/src/scr_bytes.c @@ -2240,29 +2240,26 @@ ScrStr *scr_text_decode_stream(ScrDyn *state, const ScrBytes *input, return result; } -/* Decode the next code point from an ScrStr's storage — ALWAYS valid - * UTF-8 (the string runtime never stores ill-formed sequences), so no - * validation re-runs here. */ -static uint32_t scr_bytes_next_cp(const uint8_t *in, size_t *ip) { - uint8_t b = in[*ip]; - if (b < 0x80) { - *ip += 1; - return b; - } - if ((b & 0xe0) == 0xc0) { - uint32_t cp = ((uint32_t)(b & 0x1f) << 6) | (in[*ip + 1] & 0x3f); - *ip += 2; - return cp; - } - if ((b & 0xf0) == 0xe0) { - uint32_t cp = ((uint32_t)(b & 0xf) << 12) | ((uint32_t)(in[*ip + 1] & 0x3f) << 6) | - (in[*ip + 2] & 0x3f); - *ip += 3; - return cp; - } - uint32_t cp = ((uint32_t)(b & 0x7) << 18) | ((uint32_t)(in[*ip + 1] & 0x3f) << 12) | - ((uint32_t)(in[*ip + 2] & 0x3f) << 6) | (in[*ip + 3] & 0x3f); - *ip += 4; +/* Decode the next code point from the n bytes of an ScrStr's storage. + * Storage is WTF-8, except where raw outside bytes become a string + * unchecked (HTTP header values). A byte that does not start a complete, + * well-formed sequence stands for U+0000..U+00FF, as each byte of such a + * header does in Node, and advances by one; nothing is read past n. */ +static uint32_t scr_bytes_next_cp(const uint8_t *in, size_t n, size_t *ip) { + size_t i = *ip; + uint8_t b = in[i]; + *ip = i + 1; + if (b < 0x80) return b; + size_t len = b >= 0xc2 && b <= 0xdf ? 2 : b >= 0xe0 && b <= 0xef ? 3 : b >= 0xf0 && b <= 0xf4 ? 4 : 0; + if (len == 0 || n - i < len) return b; + uint8_t lo = b == 0xe0 ? 0xa0 : b == 0xf0 ? 0x90 : 0x80; + uint8_t hi = b == 0xf4 ? 0x8f : 0xbf; + if (in[i + 1] < lo || in[i + 1] > hi) return b; + for (size_t k = 2; k < len; k++) + if ((in[i + k] & 0xc0) != 0x80) return b; + uint32_t cp = b & (0x7f >> len); + for (size_t k = 1; k < len; k++) cp = (cp << 6) | (in[i + k] & 0x3f); + *ip = i + len; return cp; } @@ -2287,7 +2284,7 @@ ScrDyn *scr_text_encode_into(const ScrDyn *source, const ScrDyn *destination) { read += ascii; if (written == limit) break; size_t end = written; - uint32_t cp = scr_bytes_next_cp((const uint8_t *)text->data, &end); + uint32_t cp = scr_bytes_next_cp((const uint8_t *)text->data, text->len, &end); if (end > dest->len) break; read += cp > 0xffff ? 2 : 1; written = end; @@ -2440,7 +2437,7 @@ ScrBytes *scr_buffer_transcode(const ScrDyn *source, const ScrDyn *from, const S } ScrStr *text = scr_str_from_utf8_lossy(input->data, len); for (size_t i = 0; i < text->len;) { - uint32_t cp = scr_bytes_next_cp((const uint8_t *)text->data, &i); + uint32_t cp = scr_bytes_next_cp((const uint8_t *)text->data, text->len, &i); written = scr_transcode_put(result->data, written, cp, to_encoding); } scr_str_release(text); @@ -2774,6 +2771,7 @@ static ScrBytes *scr_bytes_decode_b64(ScrBytesSink *sink, const uint8_t *in, siz o += 6; i += 8; } + if (i == n) break; /* the groups ended the input */ } /* Common complete groups need neither per-character branches nor state. */ if (have == 0 && n - i >= 4) { @@ -2824,7 +2822,7 @@ static ScrBytes *scr_bytes_encode_str(ScrBytesSink *sink, const ScrStr *s, const size_t o = s->ascii ? n : scr_bytes_ascii_prefix(in, n), i = o; memcpy(b->data, in, o); while (i < n) { - uint32_t cp = scr_bytes_next_cp(in, &i); + uint32_t cp = scr_bytes_next_cp(in, n, &i); if (cp > 0xffff) { cp -= 0x10000; b->data[o++] = (uint8_t)(0xd800 + (cp >> 10)); @@ -2841,7 +2839,7 @@ static ScrBytes *scr_bytes_encode_str(ScrBytesSink *sink, const ScrStr *s, const ScrBytes *b = scr_bytes_conversion_storage_in(sink, n * 2); size_t o = 0, i = 0; while (i < n) { - uint32_t cp = scr_bytes_next_cp(in, &i); + uint32_t cp = scr_bytes_next_cp(in, n, &i); if (cp > 0xffff) { cp -= 0x10000; uint32_t hi = 0xd800 + (cp >> 10), lo = 0xdc00 + (cp & 0x3ff); @@ -3745,7 +3743,7 @@ double scr_bytes_write_str(ScrBytes *b, const ScrStr *s, const ScrStr *enc, continue; } } - uint32_t cp = scr_bytes_next_cp((const uint8_t *)s->data, &i); + uint32_t cp = scr_bytes_next_cp((const uint8_t *)s->data, s->len, &i); uint32_t first = cp, second = 0; if (cp > 0xffff) { cp -= 0x10000; diff --git a/packages/runtime/src/scr_cipher.c b/packages/runtime/src/scr_cipher.c index a2db11434..33780fdfc 100644 --- a/packages/runtime/src/scr_cipher.c +++ b/packages/runtime/src/scr_cipher.c @@ -12,6 +12,7 @@ * compared in constant time. A cipher used as a stream reads and writes * through its own native Transform, which the stream unit builds on first * use, as Node's LazyTransform is. All arguments are borrowed. */ +#include "scr_cipher_names.h" #include "scr_runtime.h" #include @@ -114,52 +115,12 @@ static const ScCipherSpec sc_specs[] = { static const struct { const char *name; unsigned iv_len; -} sc_legacy[] = { - {"des", 8}, {"des-cbc", 8}, {"des-ecb", 0}, {"des-cfb", 8}, {"des-cfb1", 8}, {"des-cfb8", 8}, - {"des-ofb", 8}, {"desx", 8}, {"desx-cbc", 8}, {"bf", 8}, {"bf-cbc", 8}, {"bf-ecb", 0}, - {"bf-cfb", 8}, {"bf-ofb", 8}, {"blowfish", 8}, {"cast", 8}, {"cast-cbc", 8}, {"cast5-cbc", 8}, - {"cast5-ecb", 0}, {"cast5-cfb", 8}, {"cast5-ofb", 8}, {"rc2", 8}, {"rc2-cbc", 8}, {"rc2-ecb", 0}, - {"rc2-cfb", 8}, {"rc2-ofb", 8}, {"rc2-40-cbc", 8}, {"rc2-64-cbc", 8}, {"rc4", 0}, {"rc4-40", 0}, - {"rc4-hmac-md5", 0}, {"seed", 16}, {"seed-cbc", 16}, {"seed-ecb", 0}, {"seed-cfb", 16}, - {"seed-ofb", 16}, {"idea", 8}, {"idea-cbc", 8}, {"idea-ecb", 0}, {"idea-cfb", 8}, {"idea-ofb", 8}, -}; +} sc_legacy[] = {SCR_CIPHER_LEGACY_ENTRIES}; /* crypto.getCiphers() on Node 24.22 (OpenSSL 3.5). Names in this list that * have no implementation throw a coded SC2020 error instead of Node's * ERR_CRYPTO_UNKNOWN_CIPHER. */ -static const char *const sc_known[] = { - "aes-128-cbc", "aes-128-cbc-cts", "aes-128-ccm", "aes-128-cfb", "aes-128-cfb1", "aes-128-cfb8", - "aes-128-ctr", "aes-128-ecb", "aes-128-gcm", "aes-128-gcm-siv", "aes-128-ocb", "aes-128-ofb", - "aes-128-siv", "aes-128-wrap", "aes-128-wrap-inv", "aes-128-wrap-pad", "aes-128-wrap-pad-inv", - "aes-128-xts", "aes-192-cbc", "aes-192-cbc-cts", "aes-192-ccm", "aes-192-cfb", "aes-192-cfb1", - "aes-192-cfb8", "aes-192-ctr", "aes-192-ecb", "aes-192-gcm", "aes-192-gcm-siv", "aes-192-ocb", - "aes-192-ofb", "aes-192-siv", "aes-192-wrap", "aes-192-wrap-inv", "aes-192-wrap-pad", - "aes-192-wrap-pad-inv", "aes-256-cbc", "aes-256-cbc-cts", "aes-256-ccm", "aes-256-cfb", - "aes-256-cfb1", "aes-256-cfb8", "aes-256-ctr", "aes-256-ecb", "aes-256-gcm", "aes-256-gcm-siv", - "aes-256-ocb", "aes-256-ofb", "aes-256-siv", "aes-256-wrap", "aes-256-wrap-inv", - "aes-256-wrap-pad", "aes-256-wrap-pad-inv", "aes-256-xts", "aes128", "aes128-wrap", - "aes128-wrap-inv", "aes128-wrap-pad", "aes128-wrap-pad-inv", "aes192", "aes192-wrap", - "aes192-wrap-inv", "aes192-wrap-pad", "aes192-wrap-pad-inv", "aes256", "aes256-wrap", - "aes256-wrap-inv", "aes256-wrap-pad", "aes256-wrap-pad-inv", "aria-128-cbc", "aria-128-ccm", - "aria-128-cfb", "aria-128-cfb1", "aria-128-cfb8", "aria-128-ctr", "aria-128-ecb", "aria-128-gcm", - "aria-128-ofb", "aria-192-cbc", "aria-192-ccm", "aria-192-cfb", "aria-192-cfb1", "aria-192-cfb8", - "aria-192-ctr", "aria-192-ecb", "aria-192-gcm", "aria-192-ofb", "aria-256-cbc", "aria-256-ccm", - "aria-256-cfb", "aria-256-cfb1", "aria-256-cfb8", "aria-256-ctr", "aria-256-ecb", "aria-256-gcm", - "aria-256-ofb", "aria128", "aria192", "aria256", "camellia-128-cbc", "camellia-128-cbc-cts", - "camellia-128-cfb", "camellia-128-cfb1", "camellia-128-cfb8", "camellia-128-ctr", - "camellia-128-ecb", "camellia-128-ofb", "camellia-192-cbc", "camellia-192-cbc-cts", - "camellia-192-cfb", "camellia-192-cfb1", "camellia-192-cfb8", "camellia-192-ctr", - "camellia-192-ecb", "camellia-192-ofb", "camellia-256-cbc", "camellia-256-cbc-cts", - "camellia-256-cfb", "camellia-256-cfb1", "camellia-256-cfb8", "camellia-256-ctr", - "camellia-256-ecb", "camellia-256-ofb", "camellia128", "camellia192", "camellia256", "chacha20", - "chacha20-poly1305", "des-ede", "des-ede-cbc", "des-ede-cfb", "des-ede-ecb", "des-ede-ofb", - "des-ede3", "des-ede3-cbc", "des-ede3-cfb", "des-ede3-cfb1", "des-ede3-cfb8", "des-ede3-ecb", - "des-ede3-ofb", "des3", "des3-wrap", "id-aes128-ccm", "id-aes128-gcm", "id-aes128-wrap", - "id-aes128-wrap-pad", "id-aes192-ccm", "id-aes192-gcm", "id-aes192-wrap", "id-aes192-wrap-pad", - "id-aes256-ccm", "id-aes256-gcm", "id-aes256-wrap", "id-aes256-wrap-pad", - "id-smime-alg-cms3deswrap", "sm4", "sm4-cbc", "sm4-ccm", "sm4-cfb", "sm4-cfb128", "sm4-ctr", - "sm4-ecb", "sm4-gcm", "sm4-ofb", "sm4-ofb128", "sm4-xts", -}; +static const char *const sc_known[] = {SCR_CIPHER_KNOWN_NAMES}; enum { SC_NO_TAG = -1 }; enum { SC_TAG_UNKNOWN, SC_TAG_SET_BY_USER, SC_TAG_COMPUTED }; diff --git a/packages/runtime/src/scr_cipher_names.h b/packages/runtime/src/scr_cipher_names.h new file mode 100644 index 000000000..38b31e06e --- /dev/null +++ b/packages/runtime/src/scr_cipher_names.h @@ -0,0 +1,100 @@ +/* crypto.getCiphers() on Node 24.22 (OpenSSL 3.5), the names a cipher + * option can give. The cipher and pki units each include it: names + * without an implementation there throw a coded SC2020 error instead of + * Node's ERR_CRYPTO_UNKNOWN_CIPHER. */ +#ifndef SCR_CIPHER_NAMES_H +#define SCR_CIPHER_NAMES_H + +#define SCR_CIPHER_KNOWN_NAMES \ + "aes-128-cbc", "aes-128-cbc-cts", "aes-128-ccm", "aes-128-cfb", "aes-128-cfb1", "aes-128-cfb8", \ + "aes-128-ctr", "aes-128-ecb", "aes-128-gcm", "aes-128-gcm-siv", "aes-128-ocb", "aes-128-ofb", \ + "aes-128-siv", "aes-128-wrap", "aes-128-wrap-inv", "aes-128-wrap-pad", "aes-128-wrap-pad-inv", \ + "aes-128-xts", "aes-192-cbc", "aes-192-cbc-cts", "aes-192-ccm", "aes-192-cfb", "aes-192-cfb1", \ + "aes-192-cfb8", "aes-192-ctr", "aes-192-ecb", "aes-192-gcm", "aes-192-gcm-siv", "aes-192-ocb", \ + "aes-192-ofb", "aes-192-siv", "aes-192-wrap", "aes-192-wrap-inv", "aes-192-wrap-pad", \ + "aes-192-wrap-pad-inv", "aes-256-cbc", "aes-256-cbc-cts", "aes-256-ccm", "aes-256-cfb", \ + "aes-256-cfb1", "aes-256-cfb8", "aes-256-ctr", "aes-256-ecb", "aes-256-gcm", "aes-256-gcm-siv", \ + "aes-256-ocb", "aes-256-ofb", "aes-256-siv", "aes-256-wrap", "aes-256-wrap-inv", \ + "aes-256-wrap-pad", "aes-256-wrap-pad-inv", "aes-256-xts", "aes128", "aes128-wrap", \ + "aes128-wrap-inv", "aes128-wrap-pad", "aes128-wrap-pad-inv", "aes192", "aes192-wrap", \ + "aes192-wrap-inv", "aes192-wrap-pad", "aes192-wrap-pad-inv", "aes256", "aes256-wrap", \ + "aes256-wrap-inv", "aes256-wrap-pad", "aes256-wrap-pad-inv", "aria-128-cbc", "aria-128-ccm", \ + "aria-128-cfb", "aria-128-cfb1", "aria-128-cfb8", "aria-128-ctr", "aria-128-ecb", "aria-128-gcm", \ + "aria-128-ofb", "aria-192-cbc", "aria-192-ccm", "aria-192-cfb", "aria-192-cfb1", "aria-192-cfb8", \ + "aria-192-ctr", "aria-192-ecb", "aria-192-gcm", "aria-192-ofb", "aria-256-cbc", "aria-256-ccm", \ + "aria-256-cfb", "aria-256-cfb1", "aria-256-cfb8", "aria-256-ctr", "aria-256-ecb", "aria-256-gcm", \ + "aria-256-ofb", "aria128", "aria192", "aria256", "camellia-128-cbc", "camellia-128-cbc-cts", \ + "camellia-128-cfb", "camellia-128-cfb1", "camellia-128-cfb8", "camellia-128-ctr", \ + "camellia-128-ecb", "camellia-128-ofb", "camellia-192-cbc", "camellia-192-cbc-cts", \ + "camellia-192-cfb", "camellia-192-cfb1", "camellia-192-cfb8", "camellia-192-ctr", \ + "camellia-192-ecb", "camellia-192-ofb", "camellia-256-cbc", "camellia-256-cbc-cts", \ + "camellia-256-cfb", "camellia-256-cfb1", "camellia-256-cfb8", "camellia-256-ctr", \ + "camellia-256-ecb", "camellia-256-ofb", "camellia128", "camellia192", "camellia256", "chacha20", \ + "chacha20-poly1305", "des-ede", "des-ede-cbc", "des-ede-cfb", "des-ede-ecb", "des-ede-ofb", \ + "des-ede3", "des-ede3-cbc", "des-ede3-cfb", "des-ede3-cfb1", "des-ede3-cfb8", "des-ede3-ecb", \ + "des-ede3-ofb", "des3", "des3-wrap", "id-aes128-ccm", "id-aes128-gcm", "id-aes128-wrap", \ + "id-aes128-wrap-pad", "id-aes192-ccm", "id-aes192-gcm", "id-aes192-wrap", "id-aes192-wrap-pad", \ + "id-aes256-ccm", "id-aes256-gcm", "id-aes256-wrap", "id-aes256-wrap-pad", \ + "id-smime-alg-cms3deswrap", "sm4", "sm4-cbc", "sm4-ccm", "sm4-cfb", "sm4-cfb128", "sm4-ctr", \ + "sm4-ecb", "sm4-gcm", "sm4-ofb", "sm4-ofb128", "sm4-xts", + +/* The ciphers OpenSSL 3 keeps in its legacy provider, which Node 24 does + * not load, with their IV lengths: OpenSSL knows their names, and using + * them fails with its "unsupported" error. */ +#define SCR_CIPHER_LEGACY_ENTRIES \ + {"des", 8}, {"des-cbc", 8}, {"des-ecb", 0}, {"des-cfb", 8}, {"des-cfb1", 8}, {"des-cfb8", 8}, \ + {"des-ofb", 8}, {"desx", 8}, {"desx-cbc", 8}, {"bf", 8}, {"bf-cbc", 8}, {"bf-ecb", 0}, \ + {"bf-cfb", 8}, {"bf-ofb", 8}, {"blowfish", 8}, {"cast", 8}, {"cast-cbc", 8}, {"cast5-cbc", 8}, \ + {"cast5-ecb", 0}, {"cast5-cfb", 8}, {"cast5-ofb", 8}, {"rc2", 8}, {"rc2-cbc", 8}, {"rc2-ecb", 0}, \ + {"rc2-cfb", 8}, {"rc2-ofb", 8}, {"rc2-40-cbc", 8}, {"rc2-64-cbc", 8}, {"rc4", 0}, {"rc4-40", 0}, \ + {"rc4-hmac-md5", 0}, {"seed", 16}, {"seed-cbc", 16}, {"seed-ecb", 0}, {"seed-cfb", 16}, \ + {"seed-ofb", 16}, {"idea", 8}, {"idea-cbc", 8}, {"idea-ecb", 0}, {"idea-cfb", 8}, {"idea-ofb", 8}, + +/* The OID texts OpenSSL's providers register as cipher aliases, with the + * name each fetches (crypto.getCipherInfo(oid).name on Node 24.22). */ +#define SCR_CIPHER_OID_ALIASES \ + {"2.16.840.1.101.3.4.1.1", "aes-128-ecb"}, {"2.16.840.1.101.3.4.1.2", "aes-128-cbc"}, \ + {"2.16.840.1.101.3.4.1.3", "aes-128-ofb"}, {"2.16.840.1.101.3.4.1.4", "aes-128-cfb"}, \ + {"2.16.840.1.101.3.4.1.5", "id-aes128-wrap"}, {"2.16.840.1.101.3.4.1.6", "id-aes128-gcm"}, \ + {"2.16.840.1.101.3.4.1.7", "id-aes128-ccm"}, {"2.16.840.1.101.3.4.1.8", "id-aes128-wrap-pad"}, \ + {"2.16.840.1.101.3.4.1.21", "aes-192-ecb"}, {"2.16.840.1.101.3.4.1.22", "aes-192-cbc"}, \ + {"2.16.840.1.101.3.4.1.23", "aes-192-ofb"}, {"2.16.840.1.101.3.4.1.24", "aes-192-cfb"}, \ + {"2.16.840.1.101.3.4.1.25", "id-aes192-wrap"}, {"2.16.840.1.101.3.4.1.26", "id-aes192-gcm"}, \ + {"2.16.840.1.101.3.4.1.27", "id-aes192-ccm"}, {"2.16.840.1.101.3.4.1.28", "id-aes192-wrap-pad"}, \ + {"2.16.840.1.101.3.4.1.41", "aes-256-ecb"}, {"2.16.840.1.101.3.4.1.42", "aes-256-cbc"}, \ + {"2.16.840.1.101.3.4.1.43", "aes-256-ofb"}, {"2.16.840.1.101.3.4.1.44", "aes-256-cfb"}, \ + {"2.16.840.1.101.3.4.1.45", "id-aes256-wrap"}, {"2.16.840.1.101.3.4.1.46", "id-aes256-gcm"}, \ + {"2.16.840.1.101.3.4.1.47", "id-aes256-ccm"}, {"2.16.840.1.101.3.4.1.48", "id-aes256-wrap-pad"}, \ + {"1.2.410.200046.1.1.1", "aria-128-ecb"}, {"1.2.410.200046.1.1.2", "aria-128-cbc"}, \ + {"1.2.410.200046.1.1.3", "aria-128-cfb"}, {"1.2.410.200046.1.1.4", "aria-128-ofb"}, \ + {"1.2.410.200046.1.1.5", "aria-128-ctr"}, {"1.2.410.200046.1.1.6", "aria-192-ecb"}, \ + {"1.2.410.200046.1.1.7", "aria-192-cbc"}, {"1.2.410.200046.1.1.8", "aria-192-cfb"}, \ + {"1.2.410.200046.1.1.9", "aria-192-ofb"}, {"1.2.410.200046.1.1.10", "aria-192-ctr"}, \ + {"1.2.410.200046.1.1.11", "aria-256-ecb"}, {"1.2.410.200046.1.1.12", "aria-256-cbc"}, \ + {"1.2.410.200046.1.1.13", "aria-256-cfb"}, {"1.2.410.200046.1.1.14", "aria-256-ofb"}, \ + {"1.2.410.200046.1.1.15", "aria-256-ctr"}, {"1.2.410.200046.1.1.34", "aria-128-gcm"}, \ + {"1.2.410.200046.1.1.35", "aria-192-gcm"}, {"1.2.410.200046.1.1.36", "aria-256-gcm"}, \ + {"1.2.410.200046.1.1.37", "aria-128-ccm"}, {"1.2.410.200046.1.1.38", "aria-192-ccm"}, \ + {"1.2.410.200046.1.1.39", "aria-256-ccm"}, {"0.3.4401.5.3.1.9.1", "camellia-128-ecb"}, \ + {"0.3.4401.5.3.1.9.3", "camellia-128-ofb"}, {"0.3.4401.5.3.1.9.4", "camellia-128-cfb"}, \ + {"0.3.4401.5.3.1.9.9", "camellia-128-ctr"}, {"0.3.4401.5.3.1.9.21", "camellia-192-ecb"}, \ + {"0.3.4401.5.3.1.9.23", "camellia-192-ofb"}, {"0.3.4401.5.3.1.9.24", "camellia-192-cfb"}, \ + {"0.3.4401.5.3.1.9.29", "camellia-192-ctr"}, {"0.3.4401.5.3.1.9.41", "camellia-256-ecb"}, \ + {"0.3.4401.5.3.1.9.43", "camellia-256-ofb"}, {"0.3.4401.5.3.1.9.44", "camellia-256-cfb"}, \ + {"0.3.4401.5.3.1.9.49", "camellia-256-ctr"}, {"1.2.156.10197.1.104.1", "sm4-ecb"}, \ + {"1.2.156.10197.1.104.2", "sm4-cbc"}, {"1.2.156.10197.1.104.3", "sm4-ofb"}, \ + {"1.2.156.10197.1.104.4", "sm4-cfb"}, {"1.2.156.10197.1.104.7", "sm4-ctr"}, \ + {"1.2.156.10197.1.104.8", "sm4-gcm"}, {"1.2.156.10197.1.104.9", "sm4-ccm"}, \ + {"1.2.156.10197.1.104.10", "sm4-xts"}, {"1.2.392.200011.61.1.1.1.2", "camellia-128-cbc"}, \ + {"1.2.392.200011.61.1.1.1.3", "camellia-192-cbc"}, \ + {"1.2.392.200011.61.1.1.1.4", "camellia-256-cbc"}, {"1.2.840.113549.3.2", "rc2-cbc"}, \ + {"1.2.840.113549.3.4", "rc4"}, {"1.2.840.113549.3.7", "des-ede3-cbc"}, \ + {"1.2.840.113549.1.9.16.3.6", "id-smime-alg-cms3deswrap"}, {"1.3.14.3.2.6", "des-ecb"}, \ + {"1.3.14.3.2.7", "des-cbc"}, {"1.3.14.3.2.8", "des-ofb"}, {"1.3.14.3.2.9", "des-cfb"}, \ + {"1.3.14.3.2.17", "des-ede"}, {"1.3.111.2.1619.0.1.1", "aes-128-xts"}, \ + {"1.3.111.2.1619.0.1.2", "aes-256-xts"}, {"1.3.6.1.4.1.3029.1.2", "bf-cbc"}, \ + {"1.3.6.1.4.1.188.7.1.1.2", "idea-cbc"}, {"1.2.410.200004.1.3", "seed-ecb"}, \ + {"1.2.410.200004.1.4", "seed-cbc"}, {"1.2.410.200004.1.5", "seed-cfb"}, \ + {"1.2.410.200004.1.6", "seed-ofb"}, {"1.2.840.113533.7.66.10", "cast5-cbc"}, + +#endif diff --git a/packages/runtime/src/scr_console.c b/packages/runtime/src/scr_console.c index ad49e4f6f..67d4aca15 100644 --- a/packages/runtime/src/scr_console.c +++ b/packages/runtime/src/scr_console.c @@ -148,7 +148,7 @@ static void scr_enable_utf8_console_output(void) { } #endif -void scr_runtime_abi_v44(void) {} +void scr_runtime_abi_v45(void) {} void scr_init(void) { #ifdef SCR_WORKERS diff --git a/packages/runtime/src/scr_core.h b/packages/runtime/src/scr_core.h index 8e50ec0ca..206463e18 100644 --- a/packages/runtime/src/scr_core.h +++ b/packages/runtime/src/scr_core.h @@ -81,7 +81,7 @@ void scr_init(void); /* Program objects emitted by the bundled LLVM helper reference this symbol. * Its versioned spelling makes a mismatched manual runtime link fail before * the program can start. */ -void scr_runtime_abi_v44(void); +void scr_runtime_abi_v45(void); void scr_runtime_workers_v8(void); /* ── the trap funnel (scr_console.c; scr_library.c under -DSCR_LIB) ────── diff --git a/packages/runtime/src/scr_cycle.c b/packages/runtime/src/scr_cycle.c index d08f54a21..0d9df7106 100644 --- a/packages/runtime/src/scr_cycle.c +++ b/packages/runtime/src/scr_cycle.c @@ -603,7 +603,8 @@ static SCR_TL unsigned scr_cyc_backlog_shift[SCR_CYC_NGENS]; static SCR_TL bool scr_cyc_backlog_judged[SCR_CYC_NGENS]; #endif -/* Mature objects in the last full pass's dead set (scr_cyc_adapt_growth). */ +/* Mature objects in the last pass's dead set (scr_cyc_adapt_growth, + * scr_cyc_adapt_nursery). */ #ifndef SCR_WORKERS static SCR_TL size_t scr_cyc_mature_dead = 0; #endif @@ -639,23 +640,74 @@ static SCR_TL size_t scr_cyc_death_due_live = 0; /* The nursery trigger a program sets for itself * (scr_cyc_set_nursery_threshold), 0 for the default; SCR_CYCLE_THRESHOLD - * takes precedence over it. */ + * takes precedence over it. Either pins the trigger. */ static long scr_cyc_nursery_threshold_set = 0; static SCR_TL size_t scr_cyc_nursery_threshold_cached = 0; +#define SCR_CYC_NURSERY_ADAPTIVE SIZE_MAX -static size_t scr_cyc_nursery_threshold(void) { +/* An unpinned trigger follows where cyclic garbage dies. Garbage that + * outlives the nursery window survives its nursery pass and is promoted, + * and then only a mature or full pass finds it, after walking that level's + * live candidates. So after each scheduled mature or full pass whose dead + * set held more mature and old objects than 1/SCR_CYC_NURSERY_RATIO of what + * the nursery passes since the previous one freed, the window doubles, up + * to SCR_CYC_NURSERY_MAX_SHIFT doublings (4096 candidates); after one that + * held less than a quarter of that share, it halves, down to + * SCR_CYC_NURSERY_CANDIDATES. + * `nursery_freed`: what the nursery passes freed since the last scheduled + * mature or full pass. */ +#define SCR_CYC_NURSERY_RATIO 8 +#define SCR_CYC_NURSERY_MAX_SHIFT 4 +static SCR_TL unsigned char scr_cyc_nursery_shift = 0; +static SCR_TL size_t scr_cyc_nursery_freed = 0; + +static size_t scr_cyc_nursery_pinned(void) { if (scr_cyc_nursery_threshold_cached == 0) { const char *env = scr_getenv("SCR_CYCLE_THRESHOLD"); long v = env ? strtol(env, NULL, 10) : 0; if (v <= 0) v = scr_cyc_nursery_threshold_set; - scr_cyc_nursery_threshold_cached = v > 0 ? (size_t)v : SCR_CYC_NURSERY_CANDIDATES; + scr_cyc_nursery_threshold_cached = v > 0 ? (size_t)v : SCR_CYC_NURSERY_ADAPTIVE; } return scr_cyc_nursery_threshold_cached; } +/* The pinned trigger, or the default one. The scheduled-age rule counts + * collection points in it, so the adaptation does not lengthen how long an + * older backlog waits at them. */ +static size_t scr_cyc_nursery_base(void) { + size_t pinned = scr_cyc_nursery_pinned(); + return pinned != SCR_CYC_NURSERY_ADAPTIVE ? pinned : SCR_CYC_NURSERY_CANDIDATES; +} + +static size_t scr_cyc_nursery_threshold(void) { + size_t pinned = scr_cyc_nursery_pinned(); + if (pinned != SCR_CYC_NURSERY_ADAPTIVE) return pinned; + return (size_t)SCR_CYC_NURSERY_CANDIDATES << scr_cyc_nursery_shift; +} + +/* Adapts an unpinned trigger to the mature and old objects a scheduled + * mature or full pass reclaimed; returns whether the trigger changed. */ +static SCR_CYC_SCHEDULING bool scr_cyc_adapt_nursery(size_t older_freed) { + size_t nursery_freed = scr_cyc_nursery_freed; + scr_cyc_nursery_freed = 0; + if (scr_cyc_nursery_pinned() != SCR_CYC_NURSERY_ADAPTIVE) return false; + if (older_freed * SCR_CYC_NURSERY_RATIO > nursery_freed) { + if (scr_cyc_nursery_shift == SCR_CYC_NURSERY_MAX_SHIFT) return false; + scr_cyc_nursery_shift++; + return true; + } + if (older_freed * SCR_CYC_NURSERY_RATIO * 4 < nursery_freed && scr_cyc_nursery_shift > 0) { + scr_cyc_nursery_shift--; + return true; + } + return false; +} + void scr_cyc_set_nursery_threshold(long threshold) { scr_cyc_nursery_threshold_set = threshold; scr_cyc_nursery_threshold_cached = 0; + scr_cyc_nursery_shift = 0; + scr_cyc_nursery_freed = 0; } /* The number of buffered candidates (across every generation) at which the @@ -881,7 +933,14 @@ static SCR_CYC_SCHEDULING void scr_cyc_scheduled_pass(unsigned gen, bool by_grow size_t freed = scr_cyc_pass(gen); scr_cyc_keep_base = false; size_t old_reclaimed = scr_cyc_old_freed - old_freed_before + scr_cyc_mature_dead; - if (gen == SCR_CYC_NURSERY) return; + if (gen == SCR_CYC_NURSERY) { + scr_cyc_nursery_freed += freed; + return; + } + if (scr_cyc_adapt_nursery(old_reclaimed)) { + scr_cyc_trigger = scr_cyc_buffered() + scr_cyc_nursery_threshold(); + scr_cyc_window_update(); + } if (scr_cands.n != 0) { scr_cyc_backlog_judged[gen] = true; if (freed * SCR_CYC_GROWTH_DIV >= scr_cands.n) @@ -940,7 +999,7 @@ void scr_cyc_collect_scheduled(void) { if (!scr_cyc_older_backlog()) { scr_cyc_scheduled_mature_age = 0; } else if (++scr_cyc_scheduled_mature_age - >= scr_cyc_nursery_threshold()) { + >= scr_cyc_nursery_base()) { scr_cyc_scheduled_pass(SCR_CYC_OLD, false, false); return; } @@ -1981,7 +2040,7 @@ static SCR_CYC_COMPACT size_t scr_cyc_settle(unsigned gen_limit, bool doomed) { ScrCycHdr *h = scr_cyc_hdr(obj); const ScrCycType *type = scr_cyc_type(h); if (gen_limit < SCR_CYC_OLD) scr_cyc_freed(type, gen_limit); - else scr_cyc_mature_dead += scr_cyc_gen(h) == SCR_CYC_MATURE; + scr_cyc_mature_dead += scr_cyc_gen(h) == SCR_CYC_MATURE; #ifdef SCR_YIN if (scr_cyc_gen(h) == SCR_CYC_YOUNG) scr_yin_leave(h, obj); #endif diff --git a/packages/runtime/src/scr_dyn.h b/packages/runtime/src/scr_dyn.h index 656f2b994..1004a93c2 100644 --- a/packages/runtime/src/scr_dyn.h +++ b/packages/runtime/src/scr_dyn.h @@ -1941,7 +1941,8 @@ extern void scr_dyn_listener_fire_data(ScrClosure *cb, ScrBytes *chunk); extern void scr_dyn_listener_fire_err(ScrClosure *cb, ScrStr *msg); void *scr_dyn_retain_v(void *d); -/* After a compiled fast path decremented a surviving capsule or handle. */ +/* After a compiled fast path decremented a surviving value of a kind that + * can be a cycle candidate and is not one yet (scr_json.c). */ void scr_dyn_release_survivor(ScrDyn *d); void scr_dyn_release_v(void *d); void scr_dyn_trace_v(void *d, ScrTraceVisit visit, void *ctx); @@ -1955,6 +1956,9 @@ long scr_dyn_live_count(void); * scr_json.c, which programs with dynamic-tier code link. */ /* The builtin Object.prototype node, or NULL before its first use. */ const ScrDyn *scr_dyn_tier_object_prototype(void); +/* A read site's answer for a native array's length or a native iterator's + * next (scr_js_get_probe_native): +1, or NULL. */ +ScrDyn *scr_dyn_tier_unshaped_get(const ScrDyn *receiver, const ScrStr *key); /* Shared key bytes for `key` (+1, NUL-terminated), and their release. */ char *scr_dyn_tier_key_copy(const char *key, size_t key_len); void scr_dyn_tier_key_release(char *key, size_t key_len); diff --git a/packages/runtime/src/scr_dyn_invoke.c b/packages/runtime/src/scr_dyn_invoke.c index 2fd3aa406..e31359efd 100644 --- a/packages/runtime/src/scr_dyn_invoke.c +++ b/packages/runtime/src/scr_dyn_invoke.c @@ -159,6 +159,10 @@ static void scr_dyn_display_buf(ScrJsonBuf *b, const ScrDyn *d, bool to_string) } } +void scr_jb_join_dyn(ScrJsonBuf *b, const ScrDyn *d) { + if (d->kind != SCR_DYN_UNDEF && d->kind != SCR_DYN_NULL) scr_dyn_display_buf(b, d, true); +} + ScrStr *scr_dyn_display(const ScrDyn *d) { ScrJsonBuf b; scr_jb_init(&b); diff --git a/packages/runtime/src/scr_js.h b/packages/runtime/src/scr_js.h index 3ff9becd3..5abab2053 100644 --- a/packages/runtime/src/scr_js.h +++ b/packages/runtime/src/scr_js.h @@ -235,6 +235,10 @@ ScrDyn *scr_js_load_probe(ScrDyn *receiver, const ScrStr *key, ScrJsIc *ic); * a key that the receiver and its plain prototype chain lack, and caches * that answer. A method read uses scr_js_load_probe. */ ScrDyn *scr_js_get_probe(ScrDyn *receiver, const ScrStr *key, ScrJsIc *ic); +/* scr_js_get_probe for a site whose key is `length` or `next`, which also + * answers a native array's length and a native iterator's next + * (scr_dyn_tier_unshaped_get). */ +ScrDyn *scr_js_get_probe_native(ScrDyn *receiver, const ScrStr *key, ScrJsIc *ic); /* `receiver.key = value` (value borrowed) when it replaces a writable own * data property or adds one the prototype chain lacks: true when done. */ bool scr_js_store_probe(ScrDyn *receiver, const ScrStr *key, ScrDyn *value, ScrJsIc *ic); diff --git a/packages/runtime/src/scr_js_object.c b/packages/runtime/src/scr_js_object.c index e3d0b3a54..4cee9e7e8 100644 --- a/packages/runtime/src/scr_js_object.c +++ b/packages/runtime/src/scr_js_object.c @@ -407,6 +407,11 @@ ScrDyn *scr_js_get_probe(ScrDyn *receiver, const ScrStr *key, ScrJsIc *ic) { return scr_js_probe_load(receiver, key, ic, true); } +ScrDyn *scr_js_get_probe_native(ScrDyn *receiver, const ScrStr *key, ScrJsIc *ic) { + if (receiver->kind != SCR_DYN_OBJ) return scr_dyn_tier_unshaped_get(receiver, key); + return scr_js_probe_load(receiver, key, ic, true); +} + /* ── stores ───────────────────────────────────────────────────────────── */ /* The generic store keeps a native Error's fields in sync with these keys diff --git a/packages/runtime/src/scr_json.c b/packages/runtime/src/scr_json.c index cd9eeecde..fb62bc7a4 100644 --- a/packages/runtime/src/scr_json.c +++ b/packages/runtime/src/scr_json.c @@ -96,9 +96,13 @@ static SCR_TL ScrDyn *scr_iterator_symbol; static SCR_TL ScrDyn *scr_async_iterator_symbol; static SCR_TL ScrDyn *scr_to_primitive_symbol; static bool scr_dyn_generator(const ScrDyn *value) { - return value && value->kind == SCR_DYN_TYPED_REF && - (value->v.typed_ref.type_key[0] == 'g' || value->v.typed_ref.type_key[0] == 'a') && - (!strncmp(value->v.typed_ref.type_key, "generator<", 10) || !strncmp(value->v.typed_ref.type_key, "async-generator<", 16)); + if (!value || value->kind != SCR_DYN_TYPED_REF) return false; + /* Array keys also start with 'a': the second byte sets them apart before + * any comparison. */ + const char *key = value->v.typed_ref.type_key; + size_t len = value->v.typed_ref.type_key_len; + if (key[0] == 'g') return len >= 10 && !memcmp(key, "generator<", 10); + return key[0] == 'a' && key[1] == 's' && len >= 16 && !memcmp(key, "async-generator<", 16); } static bool scr_dyn_generator_symbol(const ScrDyn *value, const ScrDyn *key) { const ScrDyn *symbol = value->v.typed_ref.type_key[0] == 'a' ? scr_async_iterator_symbol : scr_iterator_symbol; @@ -1317,9 +1321,9 @@ static void scr_dyn_gcfree(void *ptr) { scr_dyn_dispose(ptr, true); } static void scr_dyn_destroy(void *ptr) { scr_dyn_dispose(ptr, false); } /* The cycle bookkeeping of a value that survives a release. Compiled - * release fast paths decrement first and call this for the kinds whose - * candidacy depends on per-value state, matching scr_dyn_release: arrays - * and objects not yet buffered (skipped while their document is intact, + * release fast paths decrement first and call this for every kind that can + * be a candidate and is not one yet, matching scr_dyn_release: functions + * and proxies, arrays and objects (skipped while their document is intact, * ScrDynRegion), capsules and handles. */ void scr_dyn_release_survivor(ScrDyn *d) { if (d->kind == SCR_DYN_ARR || d->kind == SCR_DYN_OBJ) { @@ -1327,6 +1331,10 @@ void scr_dyn_release_survivor(ScrDyn *d) { if (!region || !region->intact) scr_cyc_released(d); return; } + if (d->kind == SCR_DYN_FUNC || d->kind == SCR_DYN_PROXY) { + scr_cyc_released(d); + return; + } if ((d->kind == SCR_DYN_TYPED_REF && (d->v.typed_ref.materialized || d->v.typed_ref.casts || d->v.typed_ref.traced || d->prototype || d->symbol_properties || d->symbol_keys)) || @@ -4344,6 +4352,7 @@ ScrDyn *scr_dyn_typed_ref_element(const ScrDyn *d, size_t index) { double state = scr_arr_state(arr, (double)index); if (state == (double)SCR_ARR_VALUE) return scr_dyn_typed_ref_array_ops(d)->get(arr, (double)index); if (state == (double)SCR_ARR_UNDEFINED) return scr_dyn_retain(scr_dyn_undefined()); + if (state == (double)SCR_ARR_FOREIGN) return scr_arr_get_foreign(arr, (double)index); return NULL; } @@ -7494,6 +7503,23 @@ static ScrDyn *scr_native_iterator_new(ScrDyn *source) { return result; } +#ifdef SCR_JS_TIER +/* A read site's answer for a receiver without a JS shape whose property is + * plain data the runtime holds directly: a native array's length (the + * generic read would build its checked view first) and a native iterator's + * `next` (the builtin the generic read returns). +1, or NULL for the slow + * path. */ +ScrDyn *scr_dyn_tier_unshaped_get(const ScrDyn *receiver, const ScrStr *key) { + if (key->len == 6 && receiver->kind == SCR_DYN_TYPED_REF && !memcmp(key->data, "length", 6) && + scr_dyn_typed_ref_array_ops(receiver) && scr_dyn_ref_is_array(receiver)) + return scr_dyn_new_num((double)((ScrArr *)receiver->v.typed_ref.ptr)->len); + if (key->len == 4 && receiver->kind == SCR_DYN_HANDLE && + receiver->v.handle.tag == SCR_DYNH_ITERATOR && !memcmp(key->data, "next", 4)) + return scr_native_iterator_get(receiver->v.handle.ptr, key->data, key->len); + return NULL; +} +#endif + static ScrDyn *scr_native_collection_iterator(ScrDyn *source, unsigned selection) { ScrDyn *result = scr_native_iterator_new(source); ((ScrNativeIterator *)result->v.handle.ptr)->selection = selection; @@ -8384,7 +8410,9 @@ ScrDyn *scr_dyn_iterator(const ScrDyn *source, const ScrStr *spell) { } /* An array whose iterator method is the builtin values(): what calling * it with the array as `this` returns, without the call. */ - if (source->kind == SCR_DYN_ARR && method == scr_iterator_functions[1]) { + if (method == scr_iterator_functions[1] && + (source->kind == SCR_DYN_ARR || + (source->kind == SCR_DYN_TYPED_REF && scr_dyn_ref_is_array(source)))) { scr_dyn_release(method); return scr_native_iterator_new((ScrDyn *)source); } @@ -14398,6 +14426,13 @@ void scr_jb_put_dyn(ScrJsonBuf *b, const ScrDyn *d) { else scr_json_replaced_write(b, d, scr_dyn_undefined(), NULL, &scr_json_compact_gap, b->seen_len); } +void scr_jb_put_dyn_element(ScrJsonBuf *b, const ScrDyn *d, double index) { + ScrDyn *value = scr_json_to_json_at(scr_dyn_retain((ScrDyn *)d), NULL, (size_t)index); + if (!value) return; + scr_jb_put_dyn(b, value); + scr_dyn_release(value); +} + /* Untyped RC adapters (box/promise/exception-cell currency). */ void *scr_dyn_retain_v(void *d) { return scr_dyn_retain((ScrDyn *)d); } void scr_dyn_release_v(void *d) { scr_dyn_release((ScrDyn *)d); } diff --git a/packages/runtime/src/scr_json.h b/packages/runtime/src/scr_json.h index a8a59b60c..c1aa5dc10 100644 --- a/packages/runtime/src/scr_json.h +++ b/packages/runtime/src/scr_json.h @@ -180,6 +180,9 @@ void scr_dyn_from_enter(const void *v); void scr_dyn_from_leave(void); void scr_jb_putc(ScrJsonBuf *b, char c); void scr_jb_puts(ScrJsonBuf *b, const char *s); +/* Array.prototype.join's conversion of one element, appended to `b`: + * nothing for undefined and null, ToString of anything else. */ +void scr_jb_join_dyn(ScrJsonBuf *b, const ScrDyn *d); /* JS JSON.stringify number rules: NaN/±Infinity → null, -0 → 0, else * shortest-roundtrip via scr_f64_to_str. */ void scr_jb_put_f64(ScrJsonBuf *b, double v); @@ -199,6 +202,10 @@ void scr_jb_put_date(ScrJsonBuf *b, const ScrDate *date); * slot's rule); containers follow JSON.stringify (toJSON, accessors, * circular structures). May leave an exception pending. */ void scr_jb_put_dyn(ScrJsonBuf *b, const ScrDyn *d); +/* scr_jb_put_dyn for array element `index` (a foreign element of a typed + * array): its toJSON runs first, with the index as its key. May leave an + * exception pending. */ +void scr_jb_put_dyn_element(ScrJsonBuf *b, const ScrDyn *d, double index); /* The entries of record `rec`'s `unknown` index signature (its overflow * map) after the record's earlier members: separators, keys and values in * JS own-key order with JSON.stringify's member rules (toJSON called with diff --git a/packages/runtime/src/scr_lib.c b/packages/runtime/src/scr_lib.c index bd5fdfda1..a9fd25915 100644 --- a/packages/runtime/src/scr_lib.c +++ b/packages/runtime/src/scr_lib.c @@ -8145,6 +8145,11 @@ static bool scr_digest_alg(const char *name, size_t len, ScrDigestAlg *out) { return false; } +bool scr_digest_known(const char *name, size_t len) { + ScrDigestAlg alg; + return scr_digest_alg(name, len, &alg); +} + /* createHash() and crypto.hash() also accept "dss1" in any case. */ static bool scr_hash_alg(const char *name, size_t len, ScrDigestAlg *out) { if (scr_digest_alg(name, len, out)) return true; diff --git a/packages/runtime/src/scr_lib.h b/packages/runtime/src/scr_lib.h index fa447ff41..7ba9e2907 100644 --- a/packages/runtime/src/scr_lib.h +++ b/packages/runtime/src/scr_lib.h @@ -63,6 +63,9 @@ bool scr_lib_fork_info(double *target, uintptr_t *read_handle, * direct startup never filters user arguments. */ bool scr_lib_should_collapse_reexec_arg(ScrStr *cmd, ScrArr *args); ScrStr *scr_process_platform(void); /* +1 interned ("darwin", "linux", ...) */ +/* Whether OpenSSL, as Node 24 links it, names a digest `name` (its names, + * aliases and object identifiers, case-insensitively). */ +bool scr_digest_known(const char *name, size_t len); ScrStr *scr_process_cwd(void); /* +1 fresh (getcwd) */ bool scr_stdio_write(int fd, const void *data, size_t len); diff --git a/packages/runtime/src/scr_mailbox.c b/packages/runtime/src/scr_mailbox.c index b2af22774..6e5678378 100644 --- a/packages/runtime/src/scr_mailbox.c +++ b/packages/runtime/src/scr_mailbox.c @@ -21,6 +21,7 @@ #include #include #ifdef __linux__ +#include #include #endif #endif @@ -321,7 +322,32 @@ void scr_mailbox_wait(ScrMailbox *mailbox, double milliseconds) { if (!(milliseconds > 0)) return; if (scr_mailbox_spin(mailbox, milliseconds)) return; scr_mailbox_lock(mailbox); - if (!mailbox->head && !mailbox->closed) { + bool waiting = !mailbox->head && !mailbox->closed; +#ifdef __linux__ + /* Linux sleeps in ppoll(2) on the wake pipe rather than on the + * condition: a pipe write wakes its reader as a synchronous wakeup, + * which the scheduler can run on the writer's CPU when the writer is + * about to block. A worker that posts its result and then waits frees + * that CPU, so the thread it woke runs there at once instead of queueing + * behind another busy thread until the scheduler preempts it. Bytes left + * by signals whose events were already taken are drained first, under + * the lock: an event posted after the unlock writes a new byte. */ + if (waiting) scr_mailbox_drain(mailbox); + scr_mailbox_unlock(mailbox); + if (!waiting) return; + struct pollfd wake = {mailbox->read_fd, POLLIN, 0}; + if (isinf(milliseconds)) (void)ppoll(&wake, 1, NULL, NULL); + else { + /* The event loop rechecks its monotonic deadlines after every wake, + * including spurious ones. Bound conversion before casting time_t. */ + if (milliseconds > 86400000) milliseconds = 86400000; + struct timespec timeout = {(time_t)(milliseconds / 1000), + (long)ceil(fmod(milliseconds, 1000) * 1000000)}; + if (timeout.tv_nsec == 1000000000) { timeout.tv_sec++; timeout.tv_nsec = 0; } + (void)ppoll(&wake, 1, &timeout, NULL); + } +#else + if (waiting) { #ifdef _WIN32 DWORD timeout = milliseconds >= INFINITE ? INFINITE : (DWORD)ceil(milliseconds); (void)SleepConditionVariableSRW(&mailbox->condition, &mailbox->lock, timeout, 0); @@ -348,6 +374,7 @@ void scr_mailbox_wait(ScrMailbox *mailbox, double milliseconds) { #endif } scr_mailbox_unlock(mailbox); +#endif } int scr_mailbox_pollfd(const ScrMailbox *mailbox) { diff --git a/packages/runtime/src/scr_net.c b/packages/runtime/src/scr_net.c index bd1233978..a940fb7f1 100644 --- a/packages/runtime/src/scr_net.c +++ b/packages/runtime/src/scr_net.c @@ -516,6 +516,16 @@ struct ScrNetSocket { /* write buffer: [whead, wlen) of wbuf is unsent */ char *wbuf; size_t whead, wlen, wcap; + /* Node's writable state.length: each write with bytes in the buffer + * counts in full until its last byte is sent (a partly sent write + * included). wq[wq_head, wq_n) holds those writes in order, each with + * the offset where its bytes end, counted from wbase, the offset of + * wbuf[0] in everything buffered since the buffer last emptied. */ + struct ScrNetPendingWrite { + size_t end, len; + } *wq; + size_t wq_head, wq_n, wq_cap; + size_t wbase, wpending; /* receive buffer: [rhead, rlen) of rbuf is arrived-but-unconsumed — * fed by paused-mode reads ('readable' consumers) and unshift; drained * FIRST by every consumer (data listeners, the parser attach, the TLS @@ -565,6 +575,11 @@ struct ScrNetSocket { ScrNetLs lookup_ls; struct ScrNetLookupEv *lookup_evs; ScrNetLs data_ls, end_ls, close_ls, err_ls, conn_ls, timeout_ls, readable_ls; + /* Writable backpressure: a user write answered false (the bytes it left + * unsent reached SCR_NET_WRITE_HWM), so 'drain' fires once the write + * buffer empties */ + ScrNetLs drain_ls; + bool need_drain; /* Flow control (pause()/resume()): user_paused holds reads OFF even * with consumers (kernel/TCP backpressure is the buffer); flowing * counts as a consumer even with no 'data' listener @@ -747,6 +762,7 @@ void scr_net_sock_release(ScrNetSocket *s) { scr_net_ls_drop(&s->conn_ls); scr_net_ls_drop(&s->timeout_ls); scr_net_ls_drop(&s->readable_ls); + scr_net_ls_drop(&s->drain_ls); scr_net_ls_drop(&s->finish_ls); scr_net_ls_drop(&s->wcb_ls); scr_net_ls_drop(&s->ready_ls); @@ -764,6 +780,7 @@ void scr_net_sock_release(ScrNetSocket *s) { if (s->lookup_evs != NULL) scr_net_lookup_ops->free(s); scr_net_tunnel_free(s->tunnel); free(s->wbuf); + free(s->wq); free(s->rbuf); if (s->pipe_dst) scr_net_sock_release(s->pipe_dst); if (s->conn_pending) scr_net_server_release(s->conn_pending); @@ -1082,13 +1099,34 @@ static void scr_net_sock_maybe_finish_write(ScrNetSocket *s) { else scr_net_sock_update_read(s); /* the drain-to-EOF mode arms here */ } +/* The write buffer emptied (or its bytes moved elsewhere): every pending + * write is complete. */ +static void scr_net_sock_wq_clear(ScrNetSocket *s) { + s->whead = s->wlen = 0; + s->wq_head = s->wq_n = 0; + s->wbase = 0; + s->wpending = 0; +} + +/* Completes the pending writes whose last byte the flush has sent. */ +static void scr_net_sock_wq_settle(ScrNetSocket *s) { + size_t sent_to = s->wbase + s->whead; + while (s->wq_head < s->wq_n && s->wq[s->wq_head].end <= sent_to) { + s->wpending -= s->wq[s->wq_head].len; + s->wq_head++; + } + if (s->wq_head == s->wq_n) s->wq_head = s->wq_n = 0; +} + /* Append bytes to the write buffer (post-immediate-write remainder, or - * everything while connecting/buffered). */ -static void scr_net_sock_buffer(ScrNetSocket *s, const char *data, size_t len) { + * everything while connecting/buffered). `counted` is the length of the + * write they belong to, which counts in full until they are sent. */ +static void scr_net_sock_buffer(ScrNetSocket *s, const char *data, size_t len, size_t counted) { if (len == 0) return; /* compact the consumed head first */ if (s->whead > 0) { memmove(s->wbuf, s->wbuf + s->whead, s->wlen - s->whead); + s->wbase += s->whead; s->wlen -= s->whead; s->whead = 0; } @@ -1101,6 +1139,63 @@ static void scr_net_sock_buffer(ScrNetSocket *s, const char *data, size_t len) { } memcpy(s->wbuf + s->wlen, data, len); s->wlen += len; + if (s->wq_n == s->wq_cap && s->wq_head > 0) { + /* reclaim the completed writes before growing: a buffer that never + * empties completes old writes while new ones arrive */ + memmove(s->wq, s->wq + s->wq_head, (s->wq_n - s->wq_head) * sizeof *s->wq); + s->wq_n -= s->wq_head; + s->wq_head = 0; + } + if (s->wq_n == s->wq_cap) { + s->wq_cap = s->wq_cap ? s->wq_cap * 2 : 8; + s->wq = realloc(s->wq, s->wq_cap * sizeof *s->wq); + if (!s->wq) scr_net_oom(); + } + s->wq[s->wq_n].end = s->wbase + s->wlen; + s->wq[s->wq_n].len = counted; + s->wq_n++; + s->wpending += counted; +} + +/* Node's net.Socket writableHighWaterMark (the Writable default, + * getDefaultHighWaterMark()). */ +#define SCR_NET_WRITE_HWM 65536 + +/* 'drain' once a false write's bytes are all out: Node's afterWrite emits + * it when needDrain is set and nothing is pending, unless the stream is + * ending or destroyed. */ +static void scr_net_sock_maybe_drain(ScrNetSocket *s) { + if (!s->need_drain || s->wlen > s->whead) return; + if (s->wr_ending || s->close_emitted || s->fd < 0) return; + s->need_drain = false; + scr_net_fire0_this(&s->drain_ls, s, SCR_DYNH_NET_SOCKET); +} + +static void scr_net_sock_write_raw(ScrNetSocket *s, const char *data, size_t len); + +/* The write half takes writes: not ending or closed, and connected, + * connecting, or waiting to dial (the agent's deferred dial, the caller + * lookup), which buffers them until the connection is established. */ +static bool scr_net_sock_write_open(ScrNetSocket *w) { + if (w->wr_ending || w->close_emitted) return false; + return w->fd >= 0 || w->dial_deferred || w->lookup_wait; +} + +/* A user write: Writable.write's answer, as Node 24's writeOrBuffer + * computes it after the socket writes. Each write with bytes still + * unsent counts in full (wpending, Node's state.length), so the answer is + * false when the pending writes reach the high-water mark, and needDrain + * is set. A write the kernel takes at once answers true whatever its + * size. A write to an ended or closed socket answers false, and so does + * one whose send fails and closes the socket. */ +static bool scr_net_sock_user_write(ScrNetSocket *s, const char *data, size_t len) { + ScrNetSocket *w = s->adopted_by != NULL ? s->adopted_by : s; + bool open = scr_net_sock_write_open(w); + scr_net_sock_write_raw(s, data, len); + if (!open || !scr_net_sock_write_open(w)) return false; + bool ok = w->wpending < SCR_NET_WRITE_HWM; + if (!ok) w->need_drain = true; + return ok; } /* The write path shared by write()/end(data)/pipe delivery. Ignored on a @@ -1136,7 +1231,7 @@ static void scr_net_sock_write_raw(ScrNetSocket *s, const char *data, size_t len (s->tops && !s->t_est)) { /* mid-connect and mid-handshake bytes buffer; the flush after * 'connect'/establishment sends them */ - scr_net_sock_buffer(s, data, len); + scr_net_sock_buffer(s, data, len, len); scr_net_sock_update_write(s); return; } @@ -1155,7 +1250,7 @@ static void scr_net_sock_write_raw(ScrNetSocket *s, const char *data, size_t len } } if ((size_t)sent < len) { - scr_net_sock_buffer(s, data + sent, len - (size_t)sent); + scr_net_sock_buffer(s, data + sent, len - (size_t)sent, len); scr_net_sock_update_write(s); } } @@ -1220,7 +1315,7 @@ void scr_net_sock_writev_native(ScrNetSocket *s, const ScrNetSlice *slices, size skip -= len; continue; } - scr_net_sock_buffer(s, (const char *)iov[i].iov_base + skip, len - skip); + scr_net_sock_buffer(s, (const char *)iov[i].iov_base + skip, len - skip, len); skip = 0; } scr_net_sock_update_write(s); @@ -1247,11 +1342,15 @@ static void scr_net_sock_flush(ScrNetSocket *s) { } if (sent > 0) scr_net_sock_touch(s); s->whead += (size_t)sent; + scr_net_sock_wq_settle(s); } if (s->wlen == s->whead) { - s->whead = s->wlen = 0; + scr_net_sock_wq_clear(s); scr_net_sock_update_write(s); scr_net_sock_maybe_finish_write(s); + /* Node's afterWrite for a write the socket completed later: 'drain' + * now, unless the write half is ending or the socket is gone */ + scr_net_sock_maybe_drain(s); } } @@ -2735,7 +2834,7 @@ ScrNetSocket *scr_net_connect_deferred(double port, ScrStr *host /*borrowed, nul * for a slot takes a pooled socket instead of dialing. */ void scr_net_sock_move_writes(ScrNetSocket *from, ScrNetSocket *to) { if (from->wlen > from->whead) scr_net_sock_write_native(to, from->wbuf + from->whead, from->wlen - from->whead); - from->whead = from->wlen = 0; + scr_net_sock_wq_clear(from); } void scr_net_sock_dial_start(ScrNetSocket *s) { @@ -3150,16 +3249,42 @@ void scr_net_lookup_answer(ScrClosure *self, bool has_err, ScrStr *msg /*borrowe /* ── the socket surface ──────────────────────────────────────────────── */ -void scr_net_sock_write_str(ScrNetSocket *s, ScrStr *data /*borrowed*/) { +bool scr_net_sock_write_str(ScrNetSocket *s, ScrStr *data /*borrowed*/) { char *owned; const char *bytes = scr_str_utf8(data, &owned); - scr_net_sock_write_raw(s, bytes, data->len); + bool ok = scr_net_sock_user_write(s, bytes, data->len); scr_str_utf8_end(owned); + return ok; } -void scr_net_sock_write_bytes(ScrNetSocket *s, ScrBytes *data /*borrowed*/) { +bool scr_net_sock_write_bytes(ScrNetSocket *s, ScrBytes *data /*borrowed*/) { SCR_BYTES_SNAPSHOT(data); - scr_net_sock_write_raw(s, (const char *)data->data, (data->len * scr_bytes_elem_size(data->elem))); + return scr_net_sock_user_write(s, (const char *)data->data, + (data->len * scr_bytes_elem_size(data->elem))); +} + +/* socket.writableLength: Node's state.length, the length of every write + * not yet completely sent. */ +double scr_net_sock_writable_length(ScrNetSocket *s) { + ScrNetSocket *w = s->adopted_by != NULL ? s->adopted_by : s; + return (double)w->wpending; +} + +/* socket.writableNeedDrain: a write answered false and 'drain' has not + * fired, on a socket whose write half is open (including one still + * waiting to dial). */ +bool scr_net_sock_writable_need_drain(ScrNetSocket *s) { + ScrNetSocket *w = s->adopted_by != NULL ? s->adopted_by : s; + return w->need_drain && scr_net_sock_write_open(w); +} + +/* on('drain') / once('drain'). */ +void scr_net_sock_on_drain(ScrNetSocket *s, ScrClosure *cb /*moves*/, bool once) { + if (s->close_emitted) { + scr_closure_release(cb); + return; + } + scr_net_ls_add(&s->drain_ls, cb, NULL, once); } void scr_net_sock_end(ScrNetSocket *s) { @@ -5264,7 +5389,7 @@ ScrNetSocket *scr_net_sock_adopt(ScrNetSocket *raw) { tun->send_only = true; t->tunnel = tun; } - raw->whead = raw->wlen = 0; + scr_net_sock_wq_clear(raw); raw->adopted_by = scr_net_sock_retain(t); t->adopted_from = scr_net_sock_retain(raw); scr_net_sock_register(t); @@ -5523,6 +5648,7 @@ static bool scr_net_sock_settle(ScrNetSocket *sock) { scr_net_ls_drop(&sock->conn_ls); scr_net_ls_drop(&sock->timeout_ls); scr_net_ls_drop(&sock->readable_ls); + scr_net_ls_drop(&sock->drain_ls); scr_net_ls_drop(&sock->finish_ls); scr_net_ls_drop(&sock->wcb_ls); scr_net_ls_drop(&sock->ready_ls); @@ -5860,6 +5986,7 @@ static void scr_net_cleanup_atexit(void) { scr_net_ls_drop(&s->conn_ls); scr_net_ls_drop(&s->timeout_ls); scr_net_ls_drop(&s->readable_ls); + scr_net_ls_drop(&s->drain_ls); scr_net_ls_drop(&s->lookup_ls); /* Tear down the protocol layer's parser NOW: an in-flight http * exchange (a handler that threw before res.end()) holds req/res, @@ -5984,8 +6111,7 @@ static ScrDyn *scr_net_dynh_sock_invoke(void *h, ScrDyn *self, const char *metho /* fires once when the FIN goes out — once either way */ scr_net_sock_on_finish(s, scr_dyn_listener_closure0(cb)); } else if (scr_net_dynh_name_is(name, "drain")) { - /* This surface never backpressures (write answers true) — an - * accepted, never-fired registration is the consistent answer. */ + scr_net_sock_on_drain(s, scr_dyn_listener_closure0(cb), once); } else if (scr_net_dynh_tls != NULL && name->kind == SCR_DYN_STR && scr_net_dynh_tls->on(s, name->v.str->data, cb, once)) { /* a TLS event name ('secureConnect'), handled by the TLS unit */ @@ -6008,13 +6134,13 @@ static ScrDyn *scr_net_dynh_sock_invoke(void *h, ScrDyn *self, const char *metho scr_dyn_arg_type_fail("chunk", "of type string or an instance of Buffer or Uint8Array", chunk); return NULL; } - if (chunk->kind == SCR_DYN_STR) scr_net_sock_write_str(s, chunk->v.str); - else scr_net_sock_write_bytes(s, chunk->v.bytes); + bool ok = chunk->kind == SCR_DYN_STR ? scr_net_sock_write_str(s, chunk->v.str) + : scr_net_sock_write_bytes(s, chunk->v.bytes); if (argc > 1 && args[1]->kind == SCR_DYN_FUNC) { /* write(chunk, cb): fires when the buffer drains (sweep-deferred) */ scr_net_sock_on_write_flush(s, scr_dyn_listener_closure0(args[1])); } - return scr_dyn_new_bool(true); /* backpressure is not modeled (SEMANTICS.md) */ + return scr_dyn_new_bool(ok); } if (strcmp(method, "end") == 0) { const ScrDyn *chunk = argc > 0 ? args[0] : scr_dyn_undefined(); @@ -6193,10 +6319,12 @@ static ScrDyn *scr_net_dynh_sock_get(void *h, const char *key, size_t key_len) { } } if (strcmp(key, "readable") == 0) return scr_dyn_new_bool(scr_net_sock_readable(s)); + if (strcmp(key, "writableLength") == 0) return scr_dyn_new_num(scr_net_sock_writable_length(s)); + if (strcmp(key, "writableNeedDrain") == 0) + return scr_dyn_new_bool(scr_net_sock_writable_need_drain(s)); if (strcmp(key, "writableHighWaterMark") == 0 || strcmp(key, "readableHighWaterMark") == 0) { - /* Node's default stream highWaterMark — a constant here (backpressure - * is not modeled; SEMANTICS.md) */ - return scr_dyn_new_num(16384); + /* Node's default stream highWaterMark (getDefaultHighWaterMark()) */ + return scr_dyn_new_num(SCR_NET_WRITE_HWM); } if (scr_net_dynh_tls != NULL) { ScrDyn *out = NULL; @@ -6500,10 +6628,11 @@ void scr_net_dyn_install(void) { * STR/BYTES dispatch onto the typed entries; undefined/null end()s plain * (Node accepts them); anything else throws Node's ERR_INVALID_ARG_TYPE * chunk TypeError. Both borrowed. */ -void scr_net_sock_write_dynv(ScrNetSocket *s, const ScrDyn *d) { - if (d->kind == SCR_DYN_STR) scr_net_sock_write_str(s, d->v.str); - else if (d->kind == SCR_DYN_BYTES) scr_net_sock_write_bytes(s, d->v.bytes); - else scr_dyn_arg_type_fail("chunk", "of type string or an instance of Buffer or Uint8Array", d); +bool scr_net_sock_write_dynv(ScrNetSocket *s, const ScrDyn *d) { + if (d->kind == SCR_DYN_STR) return scr_net_sock_write_str(s, d->v.str); + if (d->kind == SCR_DYN_BYTES) return scr_net_sock_write_bytes(s, d->v.bytes); + scr_dyn_arg_type_fail("chunk", "of type string or an instance of Buffer or Uint8Array", d); + return false; } void scr_net_sock_end_dynv(ScrNetSocket *s, const ScrDyn *d) { diff --git a/packages/runtime/src/scr_net.h b/packages/runtime/src/scr_net.h index 95490aaa5..6a41e26d8 100644 --- a/packages/runtime/src/scr_net.h +++ b/packages/runtime/src/scr_net.h @@ -227,15 +227,21 @@ ScrNetSocket *scr_net_connect_lookup(double port, ScrStr *host /*borrowed*/, ScrClosure *lookup /*moves*/, void *answer_fn); /* +1 */ void scr_net_lookup_answer(ScrClosure *self, bool has_err, ScrStr *msg /*borrowed, nullable*/, ScrArr *ips /*borrowed, nullable*/); -void scr_net_sock_write_str(ScrNetSocket *s, ScrStr *data /*borrowed*/); -void scr_net_sock_write_bytes(ScrNetSocket *s, ScrBytes *data /*borrowed*/); +/* write(): Node's Writable answer — false when the bytes the write left + * unsent reach the 65536-byte high-water mark, after which 'drain' fires + * when the write buffer empties; false on an ended or closed socket. */ +bool scr_net_sock_write_str(ScrNetSocket *s, ScrStr *data /*borrowed*/); +bool scr_net_sock_write_bytes(ScrNetSocket *s, ScrBytes *data /*borrowed*/); +double scr_net_sock_writable_length(ScrNetSocket *s); +bool scr_net_sock_writable_need_drain(ScrNetSocket *s); +void scr_net_sock_on_drain(ScrNetSocket *s, ScrClosure *cb /*moves*/, bool once); void scr_net_sock_end(ScrNetSocket *s); void scr_net_sock_end_str(ScrNetSocket *s, ScrStr *data /*borrowed*/); void scr_net_sock_end_bytes(ScrNetSocket *s, ScrBytes *data /*borrowed*/); /* Checked-dynamic chunk forms (an untyped JS payload into a typed * socket): STR/BYTES dispatch; end() takes undefined/null as no chunk; * anything else throws Node's ERR_INVALID_ARG_TYPE chunk TypeError. */ -void scr_net_sock_write_dynv(ScrNetSocket *s, const ScrDyn *d /*borrowed*/); +bool scr_net_sock_write_dynv(ScrNetSocket *s, const ScrDyn *d /*borrowed*/); void scr_net_sock_end_dynv(ScrNetSocket *s, const ScrDyn *d /*borrowed*/); void scr_net_sock_destroy(ScrNetSocket *s); /* Flow control (pause/resume — see the struct's flag comments), the diff --git a/packages/runtime/src/scr_pki.c b/packages/runtime/src/scr_pki.c index 72136afdb..240595220 100644 --- a/packages/runtime/src/scr_pki.c +++ b/packages/runtime/src/scr_pki.c @@ -1,7 +1,7 @@ /* node:crypto asymmetric keys and signatures over the vendored mbedTLS * public-key layer: createPrivateKey, createPublicKey, generateKeyPairSync - * and generateKeyPair for "rsa", "ec", "ed25519" and "x25519" keys, - * crypto.diffieHellman (ECDH and X25519), the PrivateKeyObject and + * and generateKeyPair for "rsa", "rsa-pss", "ec", "ed25519" and "x25519" + * keys, crypto.diffieHellman (ECDH and X25519), the PrivateKeyObject and * PublicKeyObject handles (type, asymmetricKeyType, * asymmetricKeyDetails, export(), equals()), crypto.sign and crypto.verify, * and the Sign and Verify objects of createSign and createVerify. Keys read @@ -10,7 +10,8 @@ * prime256v1, secp384r1, secp521r1 and secp256k1, and the Ed25519 * arithmetic is scr_ed25519.c. Argument checks * and errors follow Node 24's lib/internal/crypto/keys.js, keygen.js and - * sig.js. Encrypted keys and other key types throw a coded SC2020 error. + * sig.js. Private keys export and import encrypted as PBES2 PKCS#8 and as + * traditional PEM. Other key types throw a coded SC2020 error. * All arguments are borrowed. */ #include "scr_runtime.h" @@ -23,6 +24,7 @@ #include "mbedtls/asn1.h" #include "mbedtls/asn1write.h" #include "mbedtls/base64.h" +#include "mbedtls/cipher.h" #include "mbedtls/bignum.h" #include "mbedtls/ecdh.h" #include "mbedtls/ecdsa.h" @@ -30,11 +32,13 @@ #include "mbedtls/md.h" #include "mbedtls/oid.h" #include "mbedtls/pk.h" +#include "mbedtls/pkcs5.h" #include "mbedtls/platform_util.h" #include "mbedtls/rsa.h" #include "mbedtls/x509_crt.h" #include "scr_pki.h" +#include "scr_cipher_names.h" /* ── errors ───────────────────────────────────────────────────────────── */ @@ -187,6 +191,13 @@ typedef struct SpKey { * bytes as given, before clamping, and the public key */ uint8_t ed_seed[32]; uint8_t ed_pub[32]; + /* An RSA-PSS key (id-RSASSA-PSS): signatures only, with PSS padding. + * With RSASSA-PSS-params it signs only with pss_md and MGF1 over + * pss_mgf1, and salts of at least pss_salt bytes. */ + bool rsa_pss; + bool pss_params; + mbedtls_md_type_t pss_md, pss_mgf1; + int pss_salt; } SpKey; static void *sp_key_retain(void *h) { @@ -245,6 +256,27 @@ static bool sp_is_ed(const SpKey *k) { return k->ed25519; } static bool sp_is_x(const SpKey *k) { return k->x25519; } +/* The PSS fields of `from` on `to`. */ +static void sp_pss_copy(SpKey *to, const SpKey *from) { + to->rsa_pss = from->rsa_pss; + to->pss_params = from->pss_params; + to->pss_md = from->pss_md; + to->pss_mgf1 = from->pss_mgf1; + to->pss_salt = from->pss_salt; +} + +/* Node's name for a digest RSASSA-PSS-params can name. */ +static const char *sp_pss_md_name(mbedtls_md_type_t md) { + switch (md) { + case MBEDTLS_MD_SHA1: return "sha1"; + case MBEDTLS_MD_SHA224: return "sha224"; + case MBEDTLS_MD_SHA256: return "sha256"; + case MBEDTLS_MD_SHA384: return "sha384"; + case MBEDTLS_MD_SHA512: return "sha512"; + default: return NULL; + } +} + /* An Ed25519 or X25519 key: 32-byte private and public keys, no details. */ static bool sp_is_okp(const SpKey *k) { return k->ed25519 || k->x25519; } @@ -341,10 +373,102 @@ static int sp_write_ed_der(const uint8_t *prefix, size_t prefix_len, const uint8 return (int)n; } +/* A digest AlgorithmIdentifier with NULL parameters, written backwards + * at *c; the written length or a negative error. */ +static int sp_write_md_alg(unsigned char **c, unsigned char *start, mbedtls_md_type_t md) { + const char *oid; + size_t oid_len; + int r, len = 0; + if ((r = mbedtls_oid_get_oid_by_md(md, &oid, &oid_len)) != 0) return r; + if ((r = mbedtls_asn1_write_null(c, start)) < 0) return r; + len += r; + if ((r = mbedtls_asn1_write_oid(c, start, oid, oid_len)) < 0) return r; + len += r; + if ((r = mbedtls_asn1_write_len(c, start, (size_t)len)) < 0) return r; + len += r; + if ((r = mbedtls_asn1_write_tag(c, start, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)) < 0) return r; + return len + r; +} + +/* An RSA-PSS key's AlgorithmIdentifier, as OpenSSL writes it: id-RSASSA-PSS + * with no parameters for a key without restrictions, or RSASSA-PSS-params + * omitting the fields at their defaults (SHA-1, MGF1 with SHA-1, salt 20). */ +static int sp_write_pss_alg(unsigned char **c, unsigned char *start, const SpKey *k) { + int r, len = 0; + if (k->pss_params) { + int params = 0; + if (k->pss_salt != 20) { + int field = 0; + if ((r = mbedtls_asn1_write_int(c, start, k->pss_salt)) < 0) return r; + field += r; + if ((r = mbedtls_asn1_write_len(c, start, (size_t)field)) < 0) return r; + field += r; + if ((r = mbedtls_asn1_write_tag(c, start, MBEDTLS_ASN1_CONTEXT_SPECIFIC | MBEDTLS_ASN1_CONSTRUCTED | 2)) < 0) + return r; + params += field + r; + } + if (k->pss_mgf1 != MBEDTLS_MD_SHA1) { + int field = 0; + if ((r = sp_write_md_alg(c, start, k->pss_mgf1)) < 0) return r; + field += r; + if ((r = mbedtls_asn1_write_oid(c, start, MBEDTLS_OID_MGF1, MBEDTLS_OID_SIZE(MBEDTLS_OID_MGF1))) < 0) + return r; + field += r; + if ((r = mbedtls_asn1_write_len(c, start, (size_t)field)) < 0) return r; + field += r; + if ((r = mbedtls_asn1_write_tag(c, start, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)) < 0) return r; + field += r; + if ((r = mbedtls_asn1_write_len(c, start, (size_t)field)) < 0) return r; + field += r; + if ((r = mbedtls_asn1_write_tag(c, start, MBEDTLS_ASN1_CONTEXT_SPECIFIC | MBEDTLS_ASN1_CONSTRUCTED | 1)) < 0) + return r; + params += field + r; + } + if (k->pss_md != MBEDTLS_MD_SHA1) { + int field = 0; + if ((r = sp_write_md_alg(c, start, k->pss_md)) < 0) return r; + field += r; + if ((r = mbedtls_asn1_write_len(c, start, (size_t)field)) < 0) return r; + field += r; + if ((r = mbedtls_asn1_write_tag(c, start, MBEDTLS_ASN1_CONTEXT_SPECIFIC | MBEDTLS_ASN1_CONSTRUCTED | 0)) < 0) + return r; + params += field + r; + } + if ((r = mbedtls_asn1_write_len(c, start, (size_t)params)) < 0) return r; + params += r; + if ((r = mbedtls_asn1_write_tag(c, start, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)) < 0) return r; + len += params + r; + } + if ((r = mbedtls_asn1_write_oid(c, start, MBEDTLS_OID_RSASSA_PSS, MBEDTLS_OID_SIZE(MBEDTLS_OID_RSASSA_PSS))) < 0) + return r; + len += r; + if ((r = mbedtls_asn1_write_len(c, start, (size_t)len)) < 0) return r; + len += r; + if ((r = mbedtls_asn1_write_tag(c, start, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)) < 0) return r; + return len + r; +} + +static int sp_write_pkcs1_public_der(SpKey *k, unsigned char *buf, size_t size); + static int sp_write_spki_der(SpKey *k, unsigned char *buf, size_t size) { if (sp_is_okp(k)) return sp_write_ed_der(sp_is_x(k) ? sp_x_spki_prefix : sp_ed_spki_prefix, 12, k->ed_pub, buf, size); - return mbedtls_pk_write_pubkey_der(&k->pk, buf, size); + if (!k->rsa_pss) return mbedtls_pk_write_pubkey_der(&k->pk, buf, size); + /* SubjectPublicKeyInfo ::= SEQUENCE { AlgorithmIdentifier, BIT STRING + * RSAPublicKey } */ + unsigned char inner[SP_DER_MAX]; + int n = sp_write_pkcs1_public_der(k, inner, sizeof inner); + if (n < 0) return n; + unsigned char *c = buf + size; + int r, len = 0; + if ((r = mbedtls_asn1_write_bitstring(&c, buf, inner + sizeof inner - n, (size_t)n * 8)) < 0) return r; + len += r; + if ((r = sp_write_pss_alg(&c, buf, k)) < 0) return r; + len += r; + if ((r = mbedtls_asn1_write_len(&c, buf, (size_t)len)) < 0) return r; + len += r; + if ((r = mbedtls_asn1_write_tag(&c, buf, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)) < 0) return r; + return len + r; } /* RSAPublicKey (PKCS#1) DER. */ @@ -414,6 +538,11 @@ static int sp_write_pkcs8_der(SpKey *k, unsigned char *buf, size_t size) { const char *oid; size_t oid_len; size_t alg_len = 0; + if (k->rsa_pss) { + if ((r = sp_write_pss_alg(&c, buf, k)) < 0) goto done; + len += (size_t)r; + goto version; + } if (sp_is_rsa(k)) { if ((r = mbedtls_asn1_write_null(&c, buf)) < 0) goto done; alg_len += (size_t)r; @@ -439,6 +568,7 @@ static int sp_write_pkcs8_der(SpKey *k, unsigned char *buf, size_t size) { alg_len += (size_t)r; len += alg_len; } +version: if ((r = mbedtls_asn1_write_int(&c, buf, 0)) < 0) goto done; len += (size_t)r; if ((r = mbedtls_asn1_write_len(&c, buf, len)) < 0) goto done; @@ -476,6 +606,622 @@ static ScrStr *sp_pem(const char *label, const unsigned char *der, size_t len) { return s; } +/* ── encrypted private keys ─────────────────────────────────────────── + * PKCS#8 EncryptedPrivateKeyInfo as OpenSSL 3 writes it (PBES2 with + * PBKDF2-HMAC-SHA256 over a 16-byte salt and 2048 iterations), and the + * traditional PEM encryption of PKCS#1 and SEC1 keys (Proc-Type and + * DEK-Info headers, the key from EVP_BytesToKey with MD5), for the ciphers + * below. Reading takes PBES2 with any HMAC-SHA PRF and these ciphers, and + * traditional PEM with the CBC ciphers through mbedTLS. */ + +typedef struct { + const char *name; /* OpenSSL's name */ + const char *pem; /* its DEK-Info name */ + mbedtls_cipher_type_t type; + const char *oid; + size_t oid_len, key_len, iv_len; + bool ecb; + /* What OpenSSL does when PKCS#8 or traditional PEM encryption takes + * the cipher, an SpKeyCipherError. */ + unsigned char pkcs8_error, traditional_error; +} SpKeyCipher; + +typedef enum { + SP_KC_OK, /* it encrypts */ + SP_KC_NO_OID, + SP_KC_EVP_CIPHER, + SP_KC_PARAMETER, + SP_KC_INPUT_LENGTH, + SP_KC_ZERO_CONTENT, + SP_KC_OPERATION, + SP_KC_PEM_CIPHER, + SP_KC_UNSUPPORTED, +} SpKeyCipherError; + +static const struct { + const char *packed, *library, *reason, *code; +} sp_key_cipher_errors[] = { + [SP_KC_NO_OID] = {"0680006C", "asn1 encoding routines", "cipher has no object identifier", + "ERR_OSSL_ASN1_CIPHER_HAS_NO_OBJECT_IDENTIFIER"}, + [SP_KC_EVP_CIPHER] = {"0300006B", "digital envelope routines", "unsupported cipher", + "ERR_OSSL_EVP_UNSUPPORTED_CIPHER"}, + [SP_KC_PARAMETER] = {"0300007A", "digital envelope routines", "cipher parameter error", + "ERR_OSSL_EVP_CIPHER_PARAMETER_ERROR"}, + [SP_KC_INPUT_LENGTH] = {"1C8000E6", "Provider routines", "invalid input length", "ERR_OSSL_INVALID_INPUT_LENGTH"}, + [SP_KC_ZERO_CONTENT] = {"068000DE", "asn1 encoding routines", "illegal zero content", + "ERR_OSSL_ASN1_ILLEGAL_ZERO_CONTENT"}, + [SP_KC_OPERATION] = {"1C800066", "Provider routines", "cipher operation failed", + "ERR_OSSL_CIPHER_OPERATION_FAILED"}, + [SP_KC_PEM_CIPHER] = {"04800071", "PEM routines", "unsupported cipher", "ERR_OSSL_PEM_UNSUPPORTED_CIPHER"}, + [SP_KC_UNSUPPORTED] = {"0308010C", "digital envelope routines", "unsupported", "ERR_OSSL_EVP_UNSUPPORTED"}, +}; + +#define SP_AES_OID(last) "\x60\x86\x48\x01\x65\x03\x04\x01" last, 9 +static const SpKeyCipher sp_key_ciphers[] = { + {"aes-128-cbc", "AES-128-CBC", MBEDTLS_CIPHER_AES_128_CBC, SP_AES_OID("\x02"), 16, 16, false}, + {"aes-192-cbc", "AES-192-CBC", MBEDTLS_CIPHER_AES_192_CBC, SP_AES_OID("\x16"), 24, 16, false}, + {"aes-256-cbc", "AES-256-CBC", MBEDTLS_CIPHER_AES_256_CBC, SP_AES_OID("\x2a"), 32, 16, false}, + {"aes-128-ecb", "AES-128-ECB", MBEDTLS_CIPHER_AES_128_ECB, SP_AES_OID("\x01"), 16, 0, true, SP_KC_OK, SP_KC_PEM_CIPHER}, + {"aes-192-ecb", "AES-192-ECB", MBEDTLS_CIPHER_AES_192_ECB, SP_AES_OID("\x15"), 24, 0, true, SP_KC_OK, SP_KC_PEM_CIPHER}, + {"aes-256-ecb", "AES-256-ECB", MBEDTLS_CIPHER_AES_256_ECB, SP_AES_OID("\x29"), 32, 0, true, SP_KC_OK, SP_KC_PEM_CIPHER}, + {"des-ede3-cbc", "DES-EDE3-CBC", MBEDTLS_CIPHER_DES_EDE3_CBC, "\x2a\x86\x48\x86\xf7\x0d\x03\x07", 8, 24, 8, false}, +}; +static const struct { + const char *alias, *name; +} sp_key_cipher_aliases[] = { + {"aes128", "aes-128-cbc"}, {"aes192", "aes-192-cbc"}, {"aes256", "aes-256-cbc"}, {"des3", "des-ede3-cbc"}, +}; + +/* The names OpenSSL 3.5 refuses to encrypt a private key with in one form + * or both, from Node 24 (the legacy-provider names fail in both). Other + * names Node knows encrypt there and are refused here. */ +#define SP_KC_FAIL(name, pkcs8, traditional) \ + {name, NULL, MBEDTLS_CIPHER_NONE, NULL, 0, 0, 0, false, SP_KC_##pkcs8, SP_KC_##traditional} +static const SpKeyCipher sp_key_cipher_failures[] = { + SP_KC_FAIL("aes-128-cbc-cts", NO_OID, OK), SP_KC_FAIL("aes-128-ccm", EVP_CIPHER, OK), + SP_KC_FAIL("aes-128-ctr", NO_OID, OK), SP_KC_FAIL("aes-128-gcm", PARAMETER, OK), + SP_KC_FAIL("aes-128-gcm-siv", NO_OID, OK), SP_KC_FAIL("aes-128-ocb", NO_OID, OK), + SP_KC_FAIL("aes-128-siv", NO_OID, PEM_CIPHER), + SP_KC_FAIL("aes-128-wrap", INPUT_LENGTH, INPUT_LENGTH), + SP_KC_FAIL("aes-128-wrap-inv", NO_OID, INPUT_LENGTH), + SP_KC_FAIL("aes-128-wrap-pad-inv", NO_OID, OK), SP_KC_FAIL("aes-128-xts", EVP_CIPHER, OK), + SP_KC_FAIL("aes-192-cbc-cts", NO_OID, OK), SP_KC_FAIL("aes-192-ccm", EVP_CIPHER, OK), + SP_KC_FAIL("aes-192-ctr", NO_OID, OK), SP_KC_FAIL("aes-192-gcm", PARAMETER, OK), + SP_KC_FAIL("aes-192-gcm-siv", NO_OID, OK), SP_KC_FAIL("aes-192-ocb", NO_OID, OK), + SP_KC_FAIL("aes-192-siv", NO_OID, PEM_CIPHER), + SP_KC_FAIL("aes-192-wrap", INPUT_LENGTH, INPUT_LENGTH), + SP_KC_FAIL("aes-192-wrap-inv", NO_OID, INPUT_LENGTH), + SP_KC_FAIL("aes-192-wrap-pad-inv", NO_OID, OK), SP_KC_FAIL("aes-256-cbc-cts", NO_OID, OK), + SP_KC_FAIL("aes-256-ccm", EVP_CIPHER, OK), SP_KC_FAIL("aes-256-ctr", NO_OID, OK), + SP_KC_FAIL("aes-256-gcm", PARAMETER, OK), SP_KC_FAIL("aes-256-gcm-siv", NO_OID, OK), + SP_KC_FAIL("aes-256-ocb", NO_OID, OK), SP_KC_FAIL("aes-256-siv", NO_OID, PEM_CIPHER), + SP_KC_FAIL("aes-256-wrap", INPUT_LENGTH, INPUT_LENGTH), + SP_KC_FAIL("aes-256-wrap-inv", NO_OID, INPUT_LENGTH), + SP_KC_FAIL("aes-256-wrap-pad-inv", NO_OID, OK), SP_KC_FAIL("aes-256-xts", EVP_CIPHER, OK), + SP_KC_FAIL("aes128-wrap", INPUT_LENGTH, INPUT_LENGTH), + SP_KC_FAIL("aes128-wrap-inv", NO_OID, INPUT_LENGTH), + SP_KC_FAIL("aes128-wrap-pad-inv", NO_OID, OK), + SP_KC_FAIL("aes192-wrap", INPUT_LENGTH, INPUT_LENGTH), + SP_KC_FAIL("aes192-wrap-inv", NO_OID, INPUT_LENGTH), + SP_KC_FAIL("aes192-wrap-pad-inv", NO_OID, OK), + SP_KC_FAIL("aes256-wrap", INPUT_LENGTH, INPUT_LENGTH), + SP_KC_FAIL("aes256-wrap-inv", NO_OID, INPUT_LENGTH), + SP_KC_FAIL("aes256-wrap-pad-inv", NO_OID, OK), SP_KC_FAIL("aria-128-ccm", EVP_CIPHER, OK), + SP_KC_FAIL("aria-128-cfb1", NO_OID, OK), SP_KC_FAIL("aria-128-cfb8", NO_OID, OK), + SP_KC_FAIL("aria-128-ecb", OK, PEM_CIPHER), SP_KC_FAIL("aria-128-gcm", PARAMETER, OK), + SP_KC_FAIL("aria-192-ccm", EVP_CIPHER, OK), SP_KC_FAIL("aria-192-cfb1", NO_OID, OK), + SP_KC_FAIL("aria-192-cfb8", NO_OID, OK), SP_KC_FAIL("aria-192-ecb", OK, PEM_CIPHER), + SP_KC_FAIL("aria-192-gcm", PARAMETER, OK), SP_KC_FAIL("aria-256-ccm", EVP_CIPHER, OK), + SP_KC_FAIL("aria-256-cfb1", NO_OID, OK), SP_KC_FAIL("aria-256-cfb8", NO_OID, OK), + SP_KC_FAIL("aria-256-ecb", OK, PEM_CIPHER), SP_KC_FAIL("aria-256-gcm", PARAMETER, OK), + SP_KC_FAIL("camellia-128-cbc-cts", NO_OID, OK), SP_KC_FAIL("camellia-128-cfb1", NO_OID, OK), + SP_KC_FAIL("camellia-128-cfb8", NO_OID, OK), SP_KC_FAIL("camellia-128-ecb", OK, PEM_CIPHER), + SP_KC_FAIL("camellia-192-cbc-cts", NO_OID, OK), SP_KC_FAIL("camellia-192-cfb1", NO_OID, OK), + SP_KC_FAIL("camellia-192-cfb8", NO_OID, OK), SP_KC_FAIL("camellia-192-ecb", OK, PEM_CIPHER), + SP_KC_FAIL("camellia-256-cbc-cts", NO_OID, OK), SP_KC_FAIL("camellia-256-cfb1", NO_OID, OK), + SP_KC_FAIL("camellia-256-cfb8", NO_OID, OK), SP_KC_FAIL("camellia-256-ecb", OK, PEM_CIPHER), + SP_KC_FAIL("chacha20", NO_OID, OK), SP_KC_FAIL("chacha20-poly1305", NO_OID, OK), + SP_KC_FAIL("des-ede", OK, PEM_CIPHER), SP_KC_FAIL("des-ede-cbc", NO_OID, OK), + SP_KC_FAIL("des-ede-cfb", NO_OID, OK), SP_KC_FAIL("des-ede-ecb", OK, PEM_CIPHER), + SP_KC_FAIL("des-ede-ofb", NO_OID, OK), SP_KC_FAIL("des-ede3", NO_OID, PEM_CIPHER), + SP_KC_FAIL("des-ede3-cfb", ZERO_CONTENT, OK), SP_KC_FAIL("des-ede3-cfb1", ZERO_CONTENT, OK), + SP_KC_FAIL("des-ede3-cfb8", ZERO_CONTENT, OK), SP_KC_FAIL("des-ede3-ecb", NO_OID, PEM_CIPHER), + SP_KC_FAIL("des-ede3-ofb", NO_OID, OK), SP_KC_FAIL("des3-wrap", OPERATION, PEM_CIPHER), + SP_KC_FAIL("id-aes128-ccm", EVP_CIPHER, OK), SP_KC_FAIL("id-aes128-gcm", PARAMETER, OK), + SP_KC_FAIL("id-aes128-wrap", INPUT_LENGTH, INPUT_LENGTH), + SP_KC_FAIL("id-aes192-ccm", EVP_CIPHER, OK), SP_KC_FAIL("id-aes192-gcm", PARAMETER, OK), + SP_KC_FAIL("id-aes192-wrap", INPUT_LENGTH, INPUT_LENGTH), + SP_KC_FAIL("id-aes256-ccm", EVP_CIPHER, OK), SP_KC_FAIL("id-aes256-gcm", PARAMETER, OK), + SP_KC_FAIL("id-aes256-wrap", INPUT_LENGTH, INPUT_LENGTH), + SP_KC_FAIL("id-smime-alg-cms3deswrap", OPERATION, PEM_CIPHER), SP_KC_FAIL("sm4-ccm", NO_OID, OK), + SP_KC_FAIL("sm4-ecb", OK, PEM_CIPHER), SP_KC_FAIL("sm4-gcm", NO_OID, OK), + SP_KC_FAIL("sm4-xts", NO_OID, OK), SP_KC_FAIL("des", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("des-cbc", UNSUPPORTED, UNSUPPORTED), SP_KC_FAIL("des-ecb", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("des-cfb", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("des-cfb1", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("des-cfb8", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("des-ofb", UNSUPPORTED, UNSUPPORTED), SP_KC_FAIL("desx", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("desx-cbc", UNSUPPORTED, UNSUPPORTED), SP_KC_FAIL("bf", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("bf-cbc", UNSUPPORTED, UNSUPPORTED), SP_KC_FAIL("bf-ecb", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("bf-cfb", UNSUPPORTED, UNSUPPORTED), SP_KC_FAIL("bf-ofb", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("blowfish", UNSUPPORTED, UNSUPPORTED), SP_KC_FAIL("cast", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("cast-cbc", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("cast5-cbc", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("cast5-ecb", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("cast5-cfb", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("cast5-ofb", UNSUPPORTED, UNSUPPORTED), SP_KC_FAIL("rc2", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("rc2-cbc", UNSUPPORTED, UNSUPPORTED), SP_KC_FAIL("rc2-ecb", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("rc2-cfb", UNSUPPORTED, UNSUPPORTED), SP_KC_FAIL("rc2-ofb", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("rc2-40-cbc", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("rc2-64-cbc", UNSUPPORTED, UNSUPPORTED), SP_KC_FAIL("rc4", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("rc4-40", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("rc4-hmac-md5", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("seed", UNSUPPORTED, UNSUPPORTED), SP_KC_FAIL("seed-cbc", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("seed-ecb", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("seed-cfb", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("seed-ofb", UNSUPPORTED, UNSUPPORTED), SP_KC_FAIL("idea", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("idea-cbc", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("idea-ecb", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("idea-cfb", UNSUPPORTED, UNSUPPORTED), + SP_KC_FAIL("idea-ofb", UNSUPPORTED, UNSUPPORTED), +}; +#undef SP_KC_FAIL + +static bool sp_ascii_eq_ci(const char *a, size_t len, const char *b) { + if (strlen(b) != len) return false; + for (size_t i = 0; i < len; i++) { + char c = a[i]; + if (c >= 'A' && c <= 'Z') c = (char)(c - 'A' + 'a'); + if (c != b[i]) return false; + } + return true; +} + +/* The cipher an export names, by a name or the OID text of one: true with + * *out set (an entry of sp_key_ciphers, or of sp_key_cipher_failures); + * false with Node's + * ERR_CRYPTO_UNKNOWN_CIPHER pending, or the SC2020 refusal for a cipher + * Node encrypts with that keys here do not. */ +static bool sp_key_cipher(const ScrStr *name, const SpKeyCipher **out) { + const char *data = name->data; + size_t len = name->len; + static const struct { + const char *oid, *name; + } oids[] = {SCR_CIPHER_OID_ALIASES}; + for (size_t i = 0; i < sizeof oids / sizeof oids[0]; i++) { + if (strlen(oids[i].oid) == len && memcmp(data, oids[i].oid, len) == 0) { + data = oids[i].name; + len = strlen(data); + } + } + const char *wanted = NULL; + for (size_t i = 0; i < sizeof sp_key_cipher_aliases / sizeof sp_key_cipher_aliases[0]; i++) + if (sp_ascii_eq_ci(data, len, sp_key_cipher_aliases[i].alias)) wanted = sp_key_cipher_aliases[i].name; + for (size_t i = 0; i < sizeof sp_key_ciphers / sizeof sp_key_ciphers[0]; i++) { + if (wanted ? strcmp(wanted, sp_key_ciphers[i].name) == 0 : sp_ascii_eq_ci(data, len, sp_key_ciphers[i].name)) { + *out = &sp_key_ciphers[i]; + return true; + } + } + for (size_t i = 0; i < sizeof sp_key_cipher_failures / sizeof sp_key_cipher_failures[0]; i++) { + if (sp_ascii_eq_ci(data, len, sp_key_cipher_failures[i].name)) { + *out = &sp_key_cipher_failures[i]; + return true; + } + } + static const char *const known[] = {SCR_CIPHER_KNOWN_NAMES}; + for (size_t i = 0; i < sizeof known / sizeof known[0]; i++) { + if (sp_ascii_eq_ci(data, len, known[i])) { + sp_not_supported("private keys encrypted with this cipher"); + return false; + } + } + sp_throw(SCR_ERR_ERROR, "ERR_CRYPTO_UNKNOWN_CIPHER", "Unknown cipher"); + return false; +} + +/* Whether encryption in the form (PKCS#8, or traditional PEM) can take + * the cipher for `len` bytes of key: false with OpenSSL's error pending, + * or the SC2020 refusal for a cipher OpenSSL encrypts with that keys here + * do not. The key wrap ciphers fail only on a length that is not a + * multiple of 8; AES key wrap then encrypts, and Node's 3DES key wrap + * corrupts its heap. */ +static bool sp_key_cipher_ready(const SpKeyCipher *ci, bool pkcs8, size_t len) { + unsigned char error = pkcs8 ? ci->pkcs8_error : ci->traditional_error; + if ((error == SP_KC_INPUT_LENGTH || error == SP_KC_OPERATION) && len % 8 == 0) error = SP_KC_OK; + if (error != SP_KC_OK) { + sp_throw_openssl(sp_key_cipher_errors[error].packed, sp_key_cipher_errors[error].library, + sp_key_cipher_errors[error].reason, sp_key_cipher_errors[error].code, true); + return false; + } + if (ci->type == MBEDTLS_CIPHER_NONE) { + sp_not_supported("private keys encrypted with this cipher"); + return false; + } + return true; +} + +/* `in` through the cipher with PKCS#7 padding, as OpenSSL's EVP layer + * pads CBC and ECB: the output (+1 malloc'd, *out_len), or NULL when + * decryption finds bad padding. */ +static unsigned char *sp_key_cipher_run(const SpKeyCipher *ci, bool encrypt, const unsigned char *key, + const unsigned char *iv, const unsigned char *in, size_t in_len, + size_t *out_len) { + const mbedtls_cipher_info_t *info = mbedtls_cipher_info_from_type(ci->type); + size_t block = mbedtls_cipher_info_get_block_size(info); + size_t padded = encrypt ? (in_len / block + 1) * block : in_len; + if (!encrypt && (in_len == 0 || in_len % block != 0)) return NULL; + unsigned char *src = malloc(padded ? padded : 1), *out = malloc(padded ? padded : 1); + if (!src || !out) scr_trap("scriptc: out of memory\n"); + memcpy(src, in, in_len); + if (encrypt) memset(src + in_len, (int)(padded - in_len), padded - in_len); + mbedtls_cipher_context_t ctx; + mbedtls_cipher_init(&ctx); + int r = mbedtls_cipher_setup(&ctx, info); + if (r == 0) r = mbedtls_cipher_setkey(&ctx, key, (int)ci->key_len * 8, encrypt ? MBEDTLS_ENCRYPT : MBEDTLS_DECRYPT); + if (r == 0 && !ci->ecb) r = mbedtls_cipher_set_padding_mode(&ctx, MBEDTLS_PADDING_NONE); + if (r == 0 && !ci->ecb) r = mbedtls_cipher_set_iv(&ctx, iv, ci->iv_len); + size_t written = 0; + for (size_t i = 0; r == 0 && i < padded; i += block) { + size_t n = 0; + r = mbedtls_cipher_update(&ctx, src + i, block, out + written, &n); + written += n; + } + if (r == 0 && !ci->ecb) { + size_t n = 0; + r = mbedtls_cipher_finish(&ctx, out + written, &n); + written += n; + } + mbedtls_cipher_free(&ctx); + mbedtls_platform_zeroize(src, padded); + free(src); + if (r == 0 && !encrypt) { + unsigned char pad = written ? out[written - 1] : 0; + bool ok = pad >= 1 && pad <= block && pad <= written; + for (size_t i = 0; ok && i < pad; i++) ok = out[written - 1 - i] == pad; + if (ok) written -= pad; + else r = -1; + } + if (r != 0) { + mbedtls_platform_zeroize(out, padded); + free(out); + return NULL; + } + *out_len = written; + return out; +} + +/* EncryptedPrivateKeyInfo DER of the PKCS#8 `plain` (+1 malloc'd, + * *out_len), as OpenSSL 3 writes it; NULL on failure. */ +static unsigned char *sp_encrypt_pkcs8(const unsigned char *plain, size_t plain_len, const SpKeyCipher *ci, + const unsigned char *pass, size_t pass_len, size_t *out_len) { + unsigned char salt[16], iv[16], key[32]; + sp_random(NULL, salt, sizeof salt); + sp_random(NULL, iv, ci->iv_len); + if (mbedtls_pkcs5_pbkdf2_hmac_ext(MBEDTLS_MD_SHA256, pass, pass_len, salt, sizeof salt, 2048, + (uint32_t)ci->key_len, key) != 0) + return NULL; + size_t enc_len; + unsigned char *enc = sp_key_cipher_run(ci, true, key, iv, plain, plain_len, &enc_len); + mbedtls_platform_zeroize(key, sizeof key); + if (!enc) return NULL; + size_t size = enc_len + 256; + unsigned char *buf = malloc(size); + if (!buf) scr_trap("scriptc: out of memory\n"); + unsigned char *c = buf + size; + int r, len = 0, w; +#define SP_W(call) do { if ((w = (call)) < 0) { r = w; goto fail; } } while (0) + SP_W(mbedtls_asn1_write_octet_string(&c, buf, enc, enc_len)); + len += w; + /* encryptionScheme: the cipher with its IV (empty for ECB) */ + int scheme = 0; + SP_W(mbedtls_asn1_write_octet_string(&c, buf, iv, ci->iv_len)); + scheme += w; + SP_W(mbedtls_asn1_write_oid(&c, buf, ci->oid, ci->oid_len)); + scheme += w; + SP_W(mbedtls_asn1_write_len(&c, buf, (size_t)scheme)); + scheme += w; + SP_W(mbedtls_asn1_write_tag(&c, buf, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)); + scheme += w; + /* keyDerivationFunc: PBKDF2 { salt, 2048, hmacWithSHA256 } */ + int kdf = 0, prf = 0; + SP_W(mbedtls_asn1_write_null(&c, buf)); + prf += w; + SP_W(mbedtls_asn1_write_oid(&c, buf, MBEDTLS_OID_HMAC_SHA256, MBEDTLS_OID_SIZE(MBEDTLS_OID_HMAC_SHA256))); + prf += w; + SP_W(mbedtls_asn1_write_len(&c, buf, (size_t)prf)); + prf += w; + SP_W(mbedtls_asn1_write_tag(&c, buf, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)); + prf += w; + kdf += prf; + SP_W(mbedtls_asn1_write_int(&c, buf, 2048)); + kdf += w; + SP_W(mbedtls_asn1_write_octet_string(&c, buf, salt, sizeof salt)); + kdf += w; + SP_W(mbedtls_asn1_write_len(&c, buf, (size_t)kdf)); + kdf += w; + SP_W(mbedtls_asn1_write_tag(&c, buf, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)); + kdf += w; + SP_W(mbedtls_asn1_write_oid(&c, buf, MBEDTLS_OID_PKCS5_PBKDF2, MBEDTLS_OID_SIZE(MBEDTLS_OID_PKCS5_PBKDF2))); + kdf += w; + SP_W(mbedtls_asn1_write_len(&c, buf, (size_t)kdf)); + kdf += w; + SP_W(mbedtls_asn1_write_tag(&c, buf, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)); + kdf += w; + int params = kdf + scheme; + SP_W(mbedtls_asn1_write_len(&c, buf, (size_t)params)); + params += w; + SP_W(mbedtls_asn1_write_tag(&c, buf, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)); + params += w; + SP_W(mbedtls_asn1_write_oid(&c, buf, MBEDTLS_OID_PKCS5_PBES2, MBEDTLS_OID_SIZE(MBEDTLS_OID_PKCS5_PBES2))); + params += w; + SP_W(mbedtls_asn1_write_len(&c, buf, (size_t)params)); + params += w; + SP_W(mbedtls_asn1_write_tag(&c, buf, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)); + params += w; + len += params; + SP_W(mbedtls_asn1_write_len(&c, buf, (size_t)len)); + len += w; + SP_W(mbedtls_asn1_write_tag(&c, buf, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)); + len += w; +#undef SP_W + memmove(buf, c, (size_t)len); + mbedtls_platform_zeroize(enc, enc_len); + free(enc); + *out_len = (size_t)len; + return buf; +fail: + (void)r; + mbedtls_platform_zeroize(enc, enc_len); + free(enc); + free(buf); + return NULL; +} + +/* OpenSSL's EVP_BytesToKey with MD5 and one round over the first 8 IV + * bytes, as traditional PEM encryption derives its key. */ +static void sp_bytes_to_key(const unsigned char *pass, size_t pass_len, const unsigned char *salt, unsigned char *key, + size_t key_len) { + const mbedtls_md_info_t *md5 = mbedtls_md_info_from_type(MBEDTLS_MD_MD5); + unsigned char digest[16]; + size_t done = 0; + mbedtls_md_context_t ctx; + mbedtls_md_init(&ctx); + mbedtls_md_setup(&ctx, md5, 0); + for (bool first = true; done < key_len; first = false) { + mbedtls_md_starts(&ctx); + if (!first) mbedtls_md_update(&ctx, digest, sizeof digest); + mbedtls_md_update(&ctx, pass, pass_len); + mbedtls_md_update(&ctx, salt, 8); + mbedtls_md_finish(&ctx, digest); + size_t n = key_len - done < sizeof digest ? key_len - done : sizeof digest; + memcpy(key + done, digest, n); + done += n; + } + mbedtls_md_free(&ctx); + mbedtls_platform_zeroize(digest, sizeof digest); +} + +/* The traditional encrypted PEM of a PKCS#1 or SEC1 key (+1), or NULL. */ +static ScrStr *sp_encrypted_pem(const char *label, const unsigned char *der, size_t len, const SpKeyCipher *ci, + const unsigned char *pass, size_t pass_len) { + unsigned char iv[16], key[32]; + sp_random(NULL, iv, ci->iv_len); + sp_bytes_to_key(pass, pass_len, iv, key, ci->key_len); + size_t enc_len; + unsigned char *enc = sp_key_cipher_run(ci, true, key, iv, der, len, &enc_len); + mbedtls_platform_zeroize(key, sizeof key); + if (!enc) return NULL; + ScrStr *body = sp_pem(label, enc, enc_len); + mbedtls_platform_zeroize(enc, enc_len); + free(enc); + /* The headers follow the BEGIN line, then a blank line. */ + const char *nl = memchr(body->data, '\n', body->len); + size_t head = (size_t)(nl - body->data) + 1; + ScrJsonBuf out; + scr_jb_init(&out); + for (size_t i = 0; i < head; i++) scr_jb_putc(&out, body->data[i]); + scr_jb_puts(&out, "Proc-Type: 4,ENCRYPTED\nDEK-Info: "); + scr_jb_puts(&out, ci->pem); + scr_jb_putc(&out, ','); + static const char hex[] = "0123456789ABCDEF"; + for (size_t i = 0; i < ci->iv_len; i++) { + scr_jb_putc(&out, hex[iv[i] >> 4]); + scr_jb_putc(&out, hex[iv[i] & 15]); + } + scr_jb_puts(&out, "\n\n"); + for (size_t i = head; i < body->len; i++) scr_jb_putc(&out, body->data[i]); + scr_str_release(body); + return scr_jb_finish(&out); +} + +/* OpenSSL's errors for an encrypted private key: the message, Node's + * code, and the error stack OpenSSL leaves for the input's form. */ +static void sp_throw_key_decrypt(const char *message, const char *library, const char *reason, const char *code, + const char *const *stack, size_t stack_len) { + ScrStr *text = scr_str_new(message, strlen(message)); + ScrError *e = scr_error_new(SCR_ERR_ERROR, text); + scr_str_release(text); + ScrDyn *list = scr_dyn_new_arr(); + for (size_t i = 0; i < stack_len; i++) { + ScrStr *entry = scr_str_new(stack[i], strlen(stack[i])); + scr_dyn_arr_push(list, scr_dyn_new_str(entry)); + scr_str_release(entry); + } + scr_error_set_prop(e, "opensslErrorStack", 17, list); + const char *keys[] = {"library", "reason", "code"}; + const char *values[] = {library, reason, code}; + for (size_t i = 0; i < 3; i++) { + if (i == 2) scr_error_set_code(e, code); + ScrStr *value = scr_str_new(values[i], strlen(values[i])); + scr_error_set_prop(e, keys[i], strlen(keys[i]), scr_dyn_new_str(value)); + scr_str_release(value); + } + scr_throw_obj(e, &scr_error_retain_v, &scr_error_release_v, scr_error_trace_arg()); +} + +static const char sp_err_interrupted[] = "error:07880109:common libcrypto routines::interrupted or cancelled"; +static const char sp_err_bad_decrypt[] = "error:1C800064:Provider routines::bad decrypt"; + +/* No passphrase for an encrypted PEM key (pkcs8: whether it is PKCS#8). */ +static void sp_throw_passphrase_missing(bool pkcs8) { + static const char *const pkcs8_stack[] = {"error:04800068:PEM routines::bad password read", sp_err_interrupted, + "error:1C80009F:Provider routines::unable to get passphrase", + sp_err_interrupted, sp_err_interrupted}; + static const char *const pem_stack[] = {"error:04800068:PEM routines::bad password read", sp_err_interrupted, + sp_err_interrupted}; + sp_throw_key_decrypt(sp_err_interrupted, "common libcrypto routines", "interrupted or cancelled", + "ERR_OSSL_CRYPTO_INTERRUPTED_OR_CANCELLED", pkcs8 ? pkcs8_stack : pem_stack, pkcs8 ? 5 : 3); +} + +/* A passphrase that does not decrypt the key: PKCS#8 in PEM, PKCS#8 in + * DER, or traditional PEM. */ +static void sp_throw_bad_decrypt(int form) { + static const char cipherfinal[] = "error:11800074:PKCS12 routines::pkcs12 cipherfinal error"; + static const char *const stacks[][4] = { + {cipherfinal, sp_err_bad_decrypt, cipherfinal, sp_err_bad_decrypt}, + {cipherfinal, sp_err_bad_decrypt}, + {"error:04800065:PEM routines::bad decrypt", sp_err_bad_decrypt}, + }; + sp_throw_key_decrypt(sp_err_bad_decrypt, "Provider routines", "bad decrypt", "ERR_OSSL_BAD_DECRYPT", stacks[form], + form == 0 ? 4 : 2); +} + +/* The cipher name of an encrypted traditional PEM key's DEK-Info header + * into `name` (NUL-terminated), or false for other input. */ +static bool sp_pem_dek_info(const unsigned char *data, size_t len, char *name, size_t cap) { + const char *text = (const char *)data; + const char *begin = NULL; + for (size_t i = 0; i + 11 <= len && !begin; i++) + if (memcmp(text + i, "-----BEGIN ", 11) == 0) begin = text + i; + if (!begin) return false; + const char *end = text + len, *line = memchr(begin, '\n', (size_t)(end - begin)); + bool encrypted = false; + while (line && ++line < end) { + const char *eol = memchr(line, '\n', (size_t)(end - line)); + size_t n = (size_t)((eol ? eol : end) - line); + if (n && line[n - 1] == '\r') n--; + if (n >= 10 && memcmp(line, "Proc-Type:", 10) == 0) { + for (size_t i = 10; i + 9 <= n; i++) + if (memcmp(line + i, "ENCRYPTED", 9) == 0) encrypted = true; + } else if (n >= 9 && memcmp(line, "DEK-Info:", 9) == 0) { + if (!encrypted) return false; + size_t i = 9; + while (i < n && line[i] == ' ') i++; + size_t o = 0; + while (i < n && line[i] != ',' && o + 1 < cap) name[o++] = line[i++]; + name[o] = 0; + return true; + } else if (n == 0 || memchr(line, ':', n) == NULL) { + return false; + } + line = eol; + } + return false; +} + +/* Before mbedTLS reads an encrypted traditional PEM key: false with + * OpenSSL's error pending for a DEK-Info cipher it does not know, a + * missing passphrase, or a legacy-provider cipher, or the SC2020 refusal + * for a cipher mbedTLS does not decrypt; true for other input. */ +static bool sp_pem_dek_check(const unsigned char *data, size_t len, const ScrBytes *pass) { + char name[64]; + if (!sp_pem_dek_info(data, len, name, sizeof name)) return true; + size_t n = strlen(name); + static const char *const known[] = {SCR_CIPHER_KNOWN_NAMES}; + static const struct { + const char *name; + unsigned iv_len; + } legacy[] = {SCR_CIPHER_LEGACY_ENTRIES}; + bool is_known = false, is_legacy = false; + for (size_t i = 0; i < sizeof known / sizeof known[0]; i++) + if (sp_ascii_eq_ci(name, n, known[i])) is_known = true; + for (size_t i = 0; i < sizeof legacy / sizeof legacy[0]; i++) + if (sp_ascii_eq_ci(name, n, legacy[i].name)) is_legacy = true; + if (!is_known && !is_legacy) { + static const char *const stack[] = {"error:04800072:PEM routines::unsupported encryption"}; + sp_throw_key_decrypt(stack[0], "PEM routines", "unsupported encryption", "ERR_OSSL_PEM_UNSUPPORTED_ENCRYPTION", + stack, 1); + return false; + } + if (!pass) { + sp_throw_passphrase_missing(false); + return false; + } + if (is_legacy) { + static const char unsupported[] = "error:0308010C:digital envelope routines::unsupported"; + static const char *const stack[] = {"error:04800065:PEM routines::bad decrypt", unsupported}; + sp_throw_key_decrypt(unsupported, "digital envelope routines", "unsupported", "ERR_OSSL_EVP_UNSUPPORTED", stack, + 2); + return false; + } + /* mbedTLS decrypts these names, as OpenSSL writes them. */ + static const char *const read[] = {"AES-128-CBC", "AES-192-CBC", "AES-256-CBC", "DES-EDE3-CBC"}; + for (size_t i = 0; i < sizeof read / sizeof read[0]; i++) + if (strcmp(name, read[i]) == 0) return true; + sp_not_supported("private keys encrypted with this cipher"); + return false; +} + +/* The PKCS#8 inside a PBES2 EncryptedPrivateKeyInfo `der` (+1 malloc'd, + * *out_len): 1 decrypted, 0 another scheme or cipher, -1 a passphrase + * that does not decrypt it. */ +static int sp_decrypt_pkcs8(const unsigned char *der, size_t len, const unsigned char *pass, size_t pass_len, + unsigned char **out, size_t *out_len) { + unsigned char *p = (unsigned char *)der; + const unsigned char *end = der + len; + size_t n; + mbedtls_asn1_buf alg, params; + if (mbedtls_asn1_get_tag(&p, end, &n, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE) != 0) return 0; + end = p + n; + if (mbedtls_asn1_get_alg(&p, end, &alg, ¶ms) != 0 || alg.len != MBEDTLS_OID_SIZE(MBEDTLS_OID_PKCS5_PBES2) || + memcmp(alg.p, MBEDTLS_OID_PKCS5_PBES2, alg.len) != 0 || + params.tag != (MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)) + return 0; + unsigned char *q = params.p; + const unsigned char *params_end = params.p + params.len; + mbedtls_asn1_buf kdf, kdf_params, scheme, scheme_params; + if (mbedtls_asn1_get_alg(&q, params_end, &kdf, &kdf_params) != 0 || + kdf.len != MBEDTLS_OID_SIZE(MBEDTLS_OID_PKCS5_PBKDF2) || memcmp(kdf.p, MBEDTLS_OID_PKCS5_PBKDF2, kdf.len) != 0 || + mbedtls_asn1_get_alg(&q, params_end, &scheme, &scheme_params) != 0) + return 0; + const SpKeyCipher *ci = NULL; + for (size_t i = 0; i < sizeof sp_key_ciphers / sizeof sp_key_ciphers[0]; i++) + if (scheme.len == sp_key_ciphers[i].oid_len && memcmp(scheme.p, sp_key_ciphers[i].oid, scheme.len) == 0) + ci = &sp_key_ciphers[i]; + if (!ci || scheme_params.tag != MBEDTLS_ASN1_OCTET_STRING || scheme_params.len != ci->iv_len) return 0; + /* PBKDF2-params ::= SEQUENCE { salt OCTET STRING, iterationCount + * INTEGER, keyLength INTEGER OPTIONAL, prf AlgorithmIdentifier DEFAULT + * hmacWithSHA1 } */ + unsigned char *k = kdf_params.p; + const unsigned char *k_end = kdf_params.p + kdf_params.len; + size_t salt_len; + int iterations, key_len = (int)ci->key_len; + if (kdf_params.tag != (MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE) || + mbedtls_asn1_get_tag(&k, k_end, &salt_len, MBEDTLS_ASN1_OCTET_STRING) != 0) + return 0; + const unsigned char *salt = k; + k += salt_len; + if (mbedtls_asn1_get_int(&k, k_end, &iterations) != 0 || iterations < 1) return 0; + if (k < k_end && *k == MBEDTLS_ASN1_INTEGER && (mbedtls_asn1_get_int(&k, k_end, &key_len) != 0 || + key_len != (int)ci->key_len)) + return 0; + mbedtls_md_type_t prf = MBEDTLS_MD_SHA1; + if (k < k_end) { + mbedtls_asn1_buf prf_oid, prf_params; + if (mbedtls_asn1_get_alg(&k, k_end, &prf_oid, &prf_params) != 0 || mbedtls_oid_get_md_hmac(&prf_oid, &prf) != 0) + return 0; + } + if (mbedtls_asn1_get_tag(&p, end, &n, MBEDTLS_ASN1_OCTET_STRING) != 0) return 0; + unsigned char key[32]; + if (mbedtls_pkcs5_pbkdf2_hmac_ext(prf, pass, pass_len, salt, salt_len, (unsigned)iterations, + (uint32_t)ci->key_len, key) != 0) + return 0; + *out = sp_key_cipher_run(ci, false, key, scheme_params.p, p, n, out_len); + mbedtls_platform_zeroize(key, sizeof key); + return *out ? 1 : -1; +} + static ScrDyn *sp_dyn_buffer(const unsigned char *data, size_t len) { ScrBytes *bytes = scr_bytes_new(SCR_BYTES_U8, (double)len); bytes->is_buffer = true; @@ -761,12 +1507,13 @@ static SpKey *sp_public_of(SpKey *k) { return pub; } unsigned char der[SP_DER_MAX]; - int n = sp_write_spki_der(k, der, sizeof der); + int n = k->rsa_pss ? mbedtls_pk_write_pubkey_der(&k->pk, der, sizeof der) : sp_write_spki_der(k, der, sizeof der); SpKey *pub = sp_key_alloc(false); if (n < 0 || mbedtls_pk_parse_public_key(&pub->pk, der + sizeof der - n, (size_t)n) != 0) { sp_key_release(pub); return NULL; } + sp_pss_copy(pub, k); return pub; } @@ -888,7 +1635,201 @@ static SpKey *sp_ed_from_der(const unsigned char *der, size_t len, bool private_ return k; } +/* The name of a digest OID that RSASSA-PSS-params can hold and this + * runtime does not sign with, for its refusal. */ +static const char *sp_pss_other_digest(const mbedtls_asn1_buf *oid) { + /* 2.16.840.1.101.3.4.2.x, the NIST hash arc */ + static const unsigned char nist[] = {0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02}; + if (oid->len == sizeof nist + 1 && memcmp(oid->p, nist, sizeof nist) == 0) { + switch (oid->p[sizeof nist]) { + case 0x05: return "sha512-224"; + case 0x06: return "sha512-256"; + case 0x07: return "sha3-224"; + case 0x08: return "sha3-256"; + case 0x09: return "sha3-384"; + case 0x0a: return "sha3-512"; + } + } + mbedtls_md_type_t md; + if (mbedtls_oid_get_md_alg(oid, &md) == 0) { + if (md == MBEDTLS_MD_MD5) return "md5"; + if (md == MBEDTLS_MD_RIPEMD160) return "ripemd160"; + } + return "this digest"; +} + +/* RSASSA-PSS-params at *p (RFC 4055): 1 with the restrictions set on `k`; + * 0 when they are malformed, which leaves the key without restrictions, as + * OpenSSL reads a negative or oversized salt; -1 with *digest named when a + * hash or MGF1 digest is one this runtime does not sign with, so that the + * key is refused rather than imported without the restrictions it carries. */ +static int sp_pss_params_read(unsigned char **p, const unsigned char *end, SpKey *k, const char **digest) { + size_t n; + if (mbedtls_asn1_get_tag(p, end, &n, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE) != 0) return 0; + const unsigned char *params_end = *p + n; + mbedtls_md_type_t md = MBEDTLS_MD_SHA1, mgf1 = MBEDTLS_MD_SHA1; + int salt = 20; + if (*p < params_end && **p == (MBEDTLS_ASN1_CONTEXT_SPECIFIC | MBEDTLS_ASN1_CONSTRUCTED | 0)) { + mbedtls_asn1_buf oid, params; + if (mbedtls_asn1_get_tag(p, params_end, &n, MBEDTLS_ASN1_CONTEXT_SPECIFIC | MBEDTLS_ASN1_CONSTRUCTED | 0) != 0 || + mbedtls_asn1_get_alg(p, *p + n, &oid, ¶ms) != 0) + return 0; + if (mbedtls_oid_get_md_alg(&oid, &md) != 0 || !sp_pss_md_name(md)) { + *digest = sp_pss_other_digest(&oid); + return -1; + } + } + if (*p < params_end && **p == (MBEDTLS_ASN1_CONTEXT_SPECIFIC | MBEDTLS_ASN1_CONSTRUCTED | 1)) { + mbedtls_asn1_buf oid, params; + if (mbedtls_asn1_get_tag(p, params_end, &n, MBEDTLS_ASN1_CONTEXT_SPECIFIC | MBEDTLS_ASN1_CONSTRUCTED | 1) != 0 || + mbedtls_asn1_get_alg(p, *p + n, &oid, ¶ms) != 0 || oid.len != MBEDTLS_OID_SIZE(MBEDTLS_OID_MGF1) || + memcmp(oid.p, MBEDTLS_OID_MGF1, oid.len) != 0) + return 0; + /* MGF1's parameters are the hash's AlgorithmIdentifier, whose + * content starts with the hash's OID. */ + if (params.tag != (MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE)) return 0; + unsigned char *alg = params.p; + mbedtls_asn1_buf hash_oid = {.tag = MBEDTLS_ASN1_OID}; + if (mbedtls_asn1_get_tag(&alg, params.p + params.len, &hash_oid.len, MBEDTLS_ASN1_OID) != 0) return 0; + hash_oid.p = alg; + if (mbedtls_oid_get_md_alg(&hash_oid, &mgf1) != 0 || !sp_pss_md_name(mgf1)) { + *digest = sp_pss_other_digest(&hash_oid); + return -1; + } + } + if (*p < params_end && **p == (MBEDTLS_ASN1_CONTEXT_SPECIFIC | MBEDTLS_ASN1_CONSTRUCTED | 2)) { + if (mbedtls_asn1_get_tag(p, params_end, &n, MBEDTLS_ASN1_CONTEXT_SPECIFIC | MBEDTLS_ASN1_CONSTRUCTED | 2) != 0 || + mbedtls_asn1_get_int(p, *p + n, &salt) != 0 || salt < 0) + return 0; + } + *p = (unsigned char *)params_end; + k->pss_params = true; + k->pss_md = md; + k->pss_mgf1 = mgf1; + k->pss_salt = salt; + return 1; +} + +/* An RSA-PSS key from PKCS#8 or SubjectPublicKeyInfo DER whose algorithm + * is id-RSASSA-PSS; NULL when the DER is neither, or with *failed set and + * Node's decoder error pending. A private key wanted as public answers its + * public key. */ +static SpKey *sp_public_of(SpKey *k); + +static SpKey *sp_pss_from_der(const unsigned char *der, size_t len, bool private_key, bool *failed) { + *failed = false; + unsigned char *p = (unsigned char *)der; + const unsigned char *end = der + len; + size_t n; + if (mbedtls_asn1_get_tag(&p, end, &n, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE) != 0) return NULL; + end = p + n; + bool pkcs8 = p < end && *p == MBEDTLS_ASN1_INTEGER; + if (pkcs8) { + int version; + if (mbedtls_asn1_get_int(&p, end, &version) != 0 || version != 0) return NULL; + } + if (mbedtls_asn1_get_tag(&p, end, &n, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE) != 0) return NULL; + const unsigned char *alg_end = p + n; + size_t oid_len; + if (mbedtls_asn1_get_tag(&p, alg_end, &oid_len, MBEDTLS_ASN1_OID) != 0 || + oid_len != MBEDTLS_OID_SIZE(MBEDTLS_OID_RSASSA_PSS) || memcmp(p, MBEDTLS_OID_RSASSA_PSS, oid_len) != 0) + return NULL; + p += oid_len; + SpKey *k = sp_key_alloc(pkcs8); + k->rsa_pss = true; + const char *digest = NULL; + if (p < alg_end) { + int read = sp_pss_params_read(&p, alg_end, k, &digest); + if (read < 0) { + sp_key_release(k); + *failed = true; + char what[96]; + snprintf(what, sizeof what, "RSA-PSS keys restricted to %s", digest); + sp_not_supported(what); + return NULL; + } + if (read == 0) k->pss_params = false; + } + p = (unsigned char *)alg_end; + int r; + if (pkcs8) { + r = mbedtls_asn1_get_tag(&p, end, &n, MBEDTLS_ASN1_OCTET_STRING); + if (r == 0) r = mbedtls_pk_parse_key(&k->pk, p, n, NULL, 0, sp_random, NULL); + } else { + r = private_key ? -1 : mbedtls_asn1_get_bitstring_null(&p, end, &n); + if (r == 0) r = mbedtls_pk_parse_public_key(&k->pk, p, n); + } + if (r != 0 || !sp_is_rsa(k)) { + sp_key_release(k); + *failed = true; + sp_unsupported_input(); + return NULL; + } + if (pkcs8 && !private_key) { + SpKey *pub = sp_public_of(k); + sp_key_release(k); + return pub; + } + return k; +} + +/* Whether `der` is a PKCS#8 EncryptedPrivateKeyInfo: a SEQUENCE of a PKCS#5 + * or PKCS#12 password-based AlgorithmIdentifier and an OCTET STRING. */ +static bool sp_is_encrypted_pkcs8(const unsigned char *der, size_t len) { + unsigned char *p = (unsigned char *)der; + const unsigned char *end = der + len; + size_t n; + mbedtls_asn1_buf alg, params; + if (mbedtls_asn1_get_tag(&p, end, &n, MBEDTLS_ASN1_CONSTRUCTED | MBEDTLS_ASN1_SEQUENCE) != 0) return false; + end = p + n; + if (mbedtls_asn1_get_alg(&p, end, &alg, ¶ms) != 0) return false; + static const char pkcs5[] = MBEDTLS_OID_PKCS5, pkcs12[] = MBEDTLS_OID_PKCS12_PBE; + bool password = (alg.len > sizeof pkcs5 - 1 && memcmp(alg.p, pkcs5, sizeof pkcs5 - 1) == 0) || + (alg.len > sizeof pkcs12 - 1 && memcmp(alg.p, pkcs12, sizeof pkcs12 - 1) == 0); + return password && mbedtls_asn1_get_tag(&p, end, &n, MBEDTLS_ASN1_OCTET_STRING) == 0 && p + n == end; +} + +static SpKey *sp_parse_with(const unsigned char *data, size_t len, bool private_key, const ScrBytes *pass); + static SpKey *sp_parse(const unsigned char *data, size_t len, bool private_key) { + return sp_parse_with(data, len, private_key, NULL); +} + +/* sp_parse with a passphrase for an encrypted private key (NULL: none). */ +static SpKey *sp_parse_with(const unsigned char *data, size_t len, bool private_key, const ScrBytes *pass) { + { + unsigned char *decoded; + const unsigned char *der; + size_t der_len; + if (sp_der_of(data, len, &decoded, &der, &der_len) && sp_is_encrypted_pkcs8(der, der_len)) { + bool from_pem = decoded != NULL; + SpKey *k = NULL; + unsigned char *plain = NULL; + size_t plain_len = 0; + int r = 0; + if (!pass) { + if (from_pem) sp_throw_passphrase_missing(true); + else sp_throw(SCR_ERR_TYPE, "ERR_MISSING_PASSPHRASE", "Passphrase required for encrypted key"); + } else { + r = sp_decrypt_pkcs8(der, der_len, pass->data, (size_t)scr_bytes_byte_len(pass), &plain, &plain_len); + if (r < 0) sp_throw_bad_decrypt(from_pem ? 0 : 1); + } + if (r > 0) { + k = sp_parse_with(plain, plain_len, private_key, NULL); + mbedtls_platform_zeroize(plain, plain_len); + free(plain); + } + if (decoded) { + mbedtls_platform_zeroize(decoded, der_len); + free(decoded); + } + if (r == 0 && pass) sp_not_supported("private keys encrypted with this scheme or cipher"); + return k; + } else if (decoded) { + mbedtls_platform_zeroize(decoded, der_len); + free(decoded); + } + } { unsigned char *decoded; const unsigned char *der; @@ -896,6 +1837,7 @@ static SpKey *sp_parse(const unsigned char *data, size_t len, bool private_key) if (sp_der_of(data, len, &decoded, &der, &der_len)) { bool failed; SpKey *ed = sp_ed_from_der(der, der_len, private_key, &failed); + if (!ed && !failed) ed = sp_pss_from_der(der, der_len, private_key, &failed); if (decoded) { mbedtls_platform_zeroize(decoded, der_len); free(decoded); @@ -912,6 +1854,7 @@ static SpKey *sp_parse(const unsigned char *data, size_t len, bool private_key) break; } } + if (pem && !sp_pem_dek_check(data, len, pass)) return NULL; unsigned char *buf = malloc(len + 1); if (!buf) scr_trap("scriptc: out of memory\n"); memcpy(buf, data, len); @@ -919,15 +1862,19 @@ static SpKey *sp_parse(const unsigned char *data, size_t len, bool private_key) size_t n = pem ? len + 1 : len; SpKey *k = sp_key_alloc(private_key); bool ok; + const unsigned char *pwd = pass ? pass->data : NULL; + size_t pwd_len = pass ? (size_t)scr_bytes_byte_len(pass) : 0; + int private_rc = 0; if (private_key) { - ok = mbedtls_pk_parse_key(&k->pk, buf, n, NULL, 0, sp_random, NULL) == 0; + private_rc = mbedtls_pk_parse_key(&k->pk, buf, n, pwd, pwd_len, sp_random, NULL); + ok = private_rc == 0; } else { ok = mbedtls_pk_parse_public_key(&k->pk, buf, n) == 0; if (!ok) { SpKey *priv = sp_key_alloc(true); mbedtls_pk_free(&k->pk); mbedtls_pk_init(&k->pk); - if (mbedtls_pk_parse_key(&priv->pk, buf, n, NULL, 0, sp_random, NULL) == 0) { + if ((private_rc = mbedtls_pk_parse_key(&priv->pk, buf, n, pwd, pwd_len, sp_random, NULL)) == 0) { sp_key_release(k); k = sp_public_of(priv); ok = k != NULL; @@ -949,7 +1896,10 @@ static SpKey *sp_parse(const unsigned char *data, size_t len, bool private_key) free(buf); if (!ok) { sp_key_release(k); - sp_unsupported_input(); + /* An encrypted traditional PEM key mbedTLS read. */ + if (private_rc == MBEDTLS_ERR_PK_PASSWORD_REQUIRED) sp_throw_passphrase_missing(false); + else if (private_rc == MBEDTLS_ERR_PK_PASSWORD_MISMATCH) sp_throw_bad_decrypt(2); + else sp_unsupported_input(); return NULL; } if (!sp_key_supported(k)) { @@ -1011,13 +1961,16 @@ static ScrDyn *sp_create_key(ScrDyn *input, bool private_key) { const ScrDyn *format = scr_dyn_undefined(); const ScrDyn *type = scr_dyn_undefined(); const ScrDyn *encoding = NULL; + const ScrDyn *passphrase = NULL; if (sp_is_object(input) && !scr_buffer_storage_is(input)) { data = sp_opt(input, "key"); format = sp_opt(input, "format"); type = sp_opt(input, "type"); encoding = sp_opt(input, "encoding"); - if (!sp_absent(sp_opt(input, "passphrase"))) { - sp_not_supported("encrypted private keys"); + passphrase = sp_opt(input, "passphrase"); + if (!sp_absent(passphrase) && passphrase->kind != SCR_DYN_STR && passphrase->kind != SCR_DYN_BYTES && + !scr_buffer_storage_is(passphrase)) { + sp_property_invalid("key.passphrase", passphrase); return NULL; } SpKey *inner = sp_key_of(data); @@ -1055,7 +2008,17 @@ static ScrDyn *sp_create_key(ScrDyn *input, bool private_key) { scr_dyn_arg_type_fail(data == input ? "key" : "key.key", sp_key_input_types, data); return NULL; } - SpKey *k = sp_parse(bytes->data, (size_t)scr_bytes_byte_len(bytes), private_key); + ScrBytes *pass = NULL; + if (passphrase && !sp_absent(passphrase)) { + /* Node decodes a string passphrase with the key's encoding. */ + pass = sp_source_bytes(passphrase, encoding); + if (!pass) { + scr_bytes_release(bytes); + return NULL; + } + } + SpKey *k = sp_parse_with(bytes->data, (size_t)scr_bytes_byte_len(bytes), private_key, pass); + if (pass) scr_bytes_release(pass); scr_bytes_release(bytes); return k ? sp_key_box(k) : NULL; } @@ -1296,7 +2259,13 @@ static ScrDyn *sp_export_plain(SpKey *k, const ScrDyn *options) { scr_dyn_arg_type_fail("options", "of type object", options ? options : scr_dyn_undefined()); return NULL; } - if (sp_is_str(format, "jwk")) return sp_export_jwk(k); + if (sp_is_str(format, "jwk")) { + if (k->rsa_pss) { + sp_throw(SCR_ERR_ERROR, "ERR_CRYPTO_JWK_UNSUPPORTED_KEY_TYPE", "Unsupported JWK Key Type."); + return NULL; + } + return sp_export_jwk(k); + } bool pem = sp_is_str(format, "pem"); if (!pem && !sp_is_str(format, "der")) { sp_property_invalid("options.format", format); @@ -1311,11 +2280,7 @@ static ScrDyn *sp_export_plain(SpKey *k, const ScrDyn *options) { sp_property_invalid("options.type", type); return NULL; } - if (!sp_absent(sp_opt(options, "cipher")) || !sp_absent(sp_opt(options, "passphrase"))) { - sp_not_supported("encrypted private key export"); - return NULL; - } - if (pkcs1 && !sp_is_rsa(k)) { + if (pkcs1 && (!sp_is_rsa(k) || k->rsa_pss)) { sp_incompatible("pkcs1", "can only be used for RSA keys"); return NULL; } @@ -1323,6 +2288,30 @@ static ScrDyn *sp_export_plain(SpKey *k, const ScrDyn *options) { sp_incompatible("sec1", "can only be used for EC keys"); return NULL; } + /* parseKeyEncoding's cipher and passphrase checks, then the cipher. */ + const ScrDyn *cipher = sp_opt(options, "cipher"), *passphrase = sp_opt(options, "passphrase"); + const SpKeyCipher *ci = NULL; + if (k->private_key) { + bool encrypted = !sp_absent(cipher) && cipher->kind != SCR_DYN_NULL; + if (encrypted) { + if (cipher->kind != SCR_DYN_STR) { + sp_property_invalid("options.cipher", cipher); + return NULL; + } + if (!pem && (pkcs1 || sec1)) { + sp_incompatible(pkcs1 ? "pkcs1" : "sec1", "does not support encryption"); + return NULL; + } + if (passphrase->kind != SCR_DYN_STR && passphrase->kind != SCR_DYN_BYTES && !scr_buffer_storage_is(passphrase)) { + sp_property_invalid("options.passphrase", passphrase); + return NULL; + } + if (!sp_key_cipher(cipher->v.str, &ci)) return NULL; + } else if (!sp_absent(passphrase)) { + sp_property_invalid_because("options.cipher", cipher, "is required when a passphrase is specified"); + return NULL; + } + } unsigned char *der = malloc(SP_DER_MAX); if (!der) scr_trap("scriptc: out of memory\n"); int n; @@ -1343,6 +2332,26 @@ static ScrDyn *sp_export_plain(SpKey *k, const ScrDyn *options) { ScrDyn *result = NULL; if (n < 0) { sp_unsupported_input(); + } else if (ci && !sp_key_cipher_ready(ci, pkcs8, (size_t)n)) { + /* OpenSSL's error, raised once the key is encoded. */ + } else if (ci) { + const unsigned char *start = der + SP_DER_MAX - n; + ScrBytes *pass = sp_source_bytes(passphrase, sp_opt(options, "encoding")); + const unsigned char *pass_data = pass ? pass->data : (const unsigned char *)""; + size_t pass_len = pass ? (size_t)scr_bytes_byte_len(pass) : 0; + if (pkcs8) { + size_t enc_len; + unsigned char *enc = sp_encrypt_pkcs8(start, (size_t)n, ci, pass_data, pass_len, &enc_len); + if (enc) { + result = pem ? sp_dyn_str(sp_pem("ENCRYPTED PRIVATE KEY", enc, enc_len)) : sp_dyn_buffer(enc, enc_len); + free(enc); + } + } else { + ScrStr *text = sp_encrypted_pem(label, start, (size_t)n, ci, pass_data, pass_len); + if (text) result = sp_dyn_str(text); + } + if (pass) scr_bytes_release(pass); + if (!result) sp_unsupported_input(); } else { const unsigned char *start = der + SP_DER_MAX - n; result = pem ? sp_dyn_str(sp_pem(label, start, (size_t)n)) : sp_dyn_buffer(start, (size_t)n); @@ -1372,6 +2381,12 @@ static ScrDyn *sp_key_details(SpKey *k) { ScrBigInt *big = scr_bigint_from_u64(exponent); sp_set(details, "publicExponent", scr_dyn_new_bigint(big)); scr_bigint_release(big); + if (k->pss_params) { + const char *md = sp_pss_md_name(k->pss_md), *mgf1 = sp_pss_md_name(k->pss_mgf1); + sp_set(details, "hashAlgorithm", sp_dyn_str(scr_str_new(md, strlen(md)))); + sp_set(details, "mgf1HashAlgorithm", sp_dyn_str(scr_str_new(mgf1, strlen(mgf1)))); + sp_set(details, "saltLength", scr_dyn_new_num(k->pss_salt)); + } } else { const SpCurve *curve = sp_curve_of(k); sp_set(details, "namedCurve", sp_dyn_str(scr_str_new(curve->name, strlen(curve->name)))); @@ -1429,6 +2444,7 @@ static ScrDyn *sp_key_get(void *h, const char *name, size_t len) { if (len == 17 && memcmp(name, "asymmetricKeyType", 17) == 0) return sp_is_ed(k) ? sp_dyn_str(scr_str_new("ed25519", 7)) : sp_is_x(k) ? sp_dyn_str(scr_str_new("x25519", 6)) + : k->rsa_pss ? sp_dyn_str(scr_str_new("rsa-pss", 7)) : sp_is_rsa(k) ? sp_dyn_str(scr_str_new("rsa", 3)) : sp_dyn_str(scr_str_new("ec", 2)); if (len == 20 && memcmp(name, "asymmetricKeyDetails", 20) == 0) return sp_key_details(k); @@ -1640,7 +2656,7 @@ static bool sp_check_encoding_plain(const ScrDyn *enc, const char *name, const S return false; } } - if (!*refusal && encrypted) *refusal = "encrypted private key export"; + (void)encrypted; return true; } @@ -1648,10 +2664,122 @@ static bool sp_check_encoding_plain(const ScrDyn *enc, const char *name, const S typedef struct { enum { SP_GEN_RSA, SP_GEN_EC, SP_GEN_ED25519, SP_GEN_X25519 } kind; double bits, exponent; /* RSA */ + bool pss; /* an RSA-PSS key, restricted when pss_params */ + bool pss_params, md_set, mgf1_set, salt_set; + mbedtls_md_type_t md, mgf1; + int salt; const SpCurve *curve; /* EC */ } SpKeyGen; +static bool sp_digest_of(const ScrStr *name, mbedtls_md_type_t *md); + +/* A digest name of rsa-pss's options: true with *md set, or false with + * Node's ERR_CRYPTO_INVALID_DIGEST pending ("Invalid digest: x" or "Invalid + * MGF1 digest: x"), or the SC2020 refusal for a digest OpenSSL has and + * RSASSA-PSS-params here cannot name. */ +static bool sp_pss_digest(const ScrDyn *name, bool mgf1, mbedtls_md_type_t *md) { + if (sp_digest_of(name->v.str, md) && sp_pss_md_name(*md)) return true; + if (scr_digest_known(name->v.str->data, name->v.str->len)) { + sp_not_supported("generateKeyPair rsa-pss with this digest"); + return false; + } + scr_pki_throw_fmt(SCR_ERR_TYPE, "ERR_CRYPTO_INVALID_DIGEST", "Invalid %sdigest: %.*s", mgf1 ? "MGF1 " : "", + (int)name->v.str->len, name->v.str->data); + return false; +} + +/* rsa-pss's options after the modulus and exponent, in Node's order + * (lib/internal/crypto/keygen.js): saltLength, hashAlgorithm, + * mgf1HashAlgorithm, and the deprecated hash and mgf1Hash with their + * DEP0154 warnings, then the digest lookups of RsaKeyGenTraits. */ +static bool sp_pss_keygen_options(const ScrDyn *options, SpKeyGen *gen) { + const ScrDyn *salt = sp_opt(options, "saltLength"); + const ScrDyn *hash_alg = sp_opt(options, "hashAlgorithm"), *mgf1_alg = sp_opt(options, "mgf1HashAlgorithm"); + const ScrDyn *hash = sp_opt(options, "hash"), *mgf1_hash = sp_opt(options, "mgf1Hash"); + if (!sp_absent(salt)) { + if (salt->kind != SCR_DYN_NUM) { + scr_dyn_prop_type_fail("options.saltLength", "of type number", salt); + return false; + } + double x = salt->v.num; + bool integer = isfinite(x) && trunc(x) == x; + if (!integer || x < 0 || x > 2147483647.0) { + ScrStr *received = sp_received(salt, true); + scr_pki_throw_fmt(SCR_ERR_RANGE, "ERR_OUT_OF_RANGE", + "The value of \"options.saltLength\" is out of range. It must be %s. Received %.*s", + integer ? ">= 0 && <= 2147483647" : "an integer", (int)received->len, received->data); + scr_str_release(received); + return false; + } + gen->salt_set = true; + gen->salt = (int)x; + } + if (!sp_absent(hash_alg) && hash_alg->kind != SCR_DYN_STR) { + scr_dyn_prop_type_fail("options.hashAlgorithm", "of type string", hash_alg); + return false; + } + if (!sp_absent(mgf1_alg) && mgf1_alg->kind != SCR_DYN_STR) { + scr_dyn_prop_type_fail("options.mgf1HashAlgorithm", "of type string", mgf1_alg); + return false; + } + const struct { + const ScrDyn *value, *preferred; + const char *name, *preferred_name; + } deprecated[] = { + {hash, hash_alg, "options.hash", "options.hashAlgorithm"}, + {mgf1_hash, mgf1_alg, "options.mgf1Hash", "options.mgf1HashAlgorithm"}, + }; + for (size_t i = 0; i < 2; i++) { + if (sp_absent(deprecated[i].value)) continue; + ScrJsonBuf text; + scr_jb_init(&text); + scr_jb_puts(&text, "\""); + scr_jb_puts(&text, deprecated[i].name); + scr_jb_puts(&text, "\" is deprecated, use \""); + scr_jb_puts(&text, deprecated[i].preferred_name); + scr_jb_puts(&text, "\" instead."); + /* process.emitWarning: reported on the next tick, dropped under + * process.noDeprecation, and thrown from its own tick under + * process.throwDeprecation. */ + ScrStr *message = scr_jb_finish(&text); + ScrStr *code = scr_str_new("DEP0154", 7); + ScrDyn *warning = scr_warning_new(message, "DeprecationWarning", code); + scr_str_release(code); + scr_str_release(message); + scr_warning_emit_deferred(warning); + scr_dyn_release(warning); + if (deprecated[i].value->kind != SCR_DYN_STR) { + scr_dyn_prop_type_fail(deprecated[i].name, "of type string", deprecated[i].value); + return false; + } + const ScrDyn *preferred = deprecated[i].preferred; + if (preferred->kind == SCR_DYN_STR && preferred->v.str->len > 0 && + !scr_str_eq(preferred->v.str, deprecated[i].value->v.str)) { + sp_property_invalid(deprecated[i].name, deprecated[i].value); + return false; + } + } + /* `hashAlgorithm || hash`: an empty string reads as absent. */ + const ScrDyn *md = hash_alg->kind == SCR_DYN_STR && hash_alg->v.str->len > 0 ? hash_alg : hash; + const ScrDyn *mgf1 = mgf1_alg->kind == SCR_DYN_STR && mgf1_alg->v.str->len > 0 ? mgf1_alg : mgf1_hash; + if (md->kind == SCR_DYN_STR) { + if (!sp_pss_digest(md, false, &gen->md)) return false; + gen->md_set = true; + } + if (mgf1->kind == SCR_DYN_STR) { + if (!sp_pss_digest(mgf1, true, &gen->mgf1)) return false; + gen->mgf1_set = true; + } + /* OpenSSL's defaults: MGF1 over the digest and a salt of the digest's + * length once a digest is given, SHA-1 and 20 otherwise. */ + gen->pss_params = gen->md_set || gen->mgf1_set || gen->salt_set; + if (!gen->md_set) gen->md = MBEDTLS_MD_SHA1; + if (!gen->mgf1_set) gen->mgf1 = gen->md; + if (!gen->salt_set) gen->salt = gen->md_set ? (int)mbedtls_md_get_size(mbedtls_md_info_from_type(gen->md)) : 20; + return true; +} + /* createJob's checks for generateKeyPair(Sync)(type, options) in Node's * order: the type, the options object, the public and private key * encodings, then the options of the type. False with the error pending. @@ -1677,7 +2805,8 @@ static bool sp_keygen_check(const ScrDyn *type, const ScrDyn *options, SpKeyGen if (!sp_absent(exponent) && exponent->kind != SCR_DYN_NULL && !sp_check_uint32(exponent, "options.publicExponent", &gen->exponent)) return false; - if (pss && !refusal) refusal = "generateKeyPairSync for this key type"; + gen->pss = pss; + if (pss && !sp_pss_keygen_options(options, gen)) return false; } else if (sp_is_str(type, "dsa") || sp_is_str(type, "dh")) { if (!sp_check_object(options, "options")) return false; if (!refusal) refusal = "generateKeyPairSync for this key type"; @@ -1723,6 +2852,23 @@ static bool sp_keygen_check(const ScrDyn *type, const ScrDyn *options, SpKeyGen } if (!refusal) refusal = "generateKeyPairSync for this key type"; } + /* The job's private key cipher, which Node looks up once the options + * of the type are read. */ + { + const ScrDyn *enc = sp_opt(options, "privateKeyEncoding"); + const ScrDyn *cipher = sp_is_object(enc) ? sp_opt(enc, "cipher") : scr_dyn_undefined(); + const SpKeyCipher *ci; + if (cipher->kind == SCR_DYN_STR) { + if (!sp_key_cipher(cipher->v.str, &ci)) return false; + /* OpenSSL's errors for the cipher come from the export once the + * key is generated; a cipher it encrypts with and keys here do not + * is refused first. */ + bool pkcs8 = sp_is_str(sp_opt(enc, "type"), "pkcs8"); + if (ci->type == MBEDTLS_CIPHER_NONE && (pkcs8 ? ci->pkcs8_error : ci->traditional_error) == SP_KC_OK && + !refusal) + refusal = "private keys encrypted with this cipher"; + } + } if (refusal) { sp_not_supported(refusal); return false; @@ -1730,6 +2876,54 @@ static bool sp_keygen_check(const ScrDyn *type, const ScrDyn *options, SpKeyGen return gen->kind != SP_GEN_ED25519 || sp_ed() != NULL; } +/* An RSA key of `bits` bits with public exponent `exponent`, within + * OpenSSL's limits rather than mbedtls_rsa_gen_key's (even lengths from + * 1024 bits, exponents up to INT_MAX): any length from 512 bits and odd + * exponents up to 2^32 - 1. From 2048 bits OpenSSL generates two primes + * of equal length (SP 800-56B), so an odd length yields one bit less. + * The primes have (bits + 1) / 2 and bits / 2 bits, each with its top + * bits set (mbedtls_mpi_gen_prime's FIPS 186-4 bound), so their product + * has exactly `bits` bits; each prime minus one is coprime to the + * exponent. 0 on success. */ +static int sp_rsa_generate(mbedtls_rsa_context *rsa, unsigned bits, uint64_t exponent) { + if (bits >= 2048) bits &= ~1u; + mbedtls_mpi P, Q, E, N, T, G; + mbedtls_mpi_init(&P); + mbedtls_mpi_init(&Q); + mbedtls_mpi_init(&E); + mbedtls_mpi_init(&N); + mbedtls_mpi_init(&T); + mbedtls_mpi_init(&G); + unsigned char e_bytes[8]; + for (int i = 0; i < 8; i++) e_bytes[i] = (unsigned char)(exponent >> (56 - 8 * i)); + int r = mbedtls_mpi_read_binary(&E, e_bytes, sizeof e_bytes); + const int flags = MBEDTLS_MPI_GEN_PRIME_FLAG_LOW_ERR; + while (r == 0) { + r = mbedtls_mpi_gen_prime(&P, (bits + 1) / 2, flags, sp_random, NULL); + if (r == 0) r = mbedtls_mpi_sub_int(&T, &P, 1); + if (r == 0) r = mbedtls_mpi_gcd(&G, &E, &T); + if (r != 0 || mbedtls_mpi_cmp_int(&G, 1) != 0) continue; + r = mbedtls_mpi_gen_prime(&Q, bits / 2, flags, sp_random, NULL); + if (r == 0) r = mbedtls_mpi_sub_int(&T, &Q, 1); + if (r == 0) r = mbedtls_mpi_gcd(&G, &E, &T); + if (r != 0 || mbedtls_mpi_cmp_int(&G, 1) != 0 || mbedtls_mpi_cmp_mpi(&P, &Q) == 0) continue; + r = mbedtls_mpi_mul_mpi(&N, &P, &Q); + if (r == 0 && mbedtls_mpi_bitlen(&N) == bits) break; + } + /* The larger prime is p, as OpenSSL orders them. */ + if (r == 0 && mbedtls_mpi_cmp_mpi(&P, &Q) < 0) mbedtls_mpi_swap(&P, &Q); + if (r == 0) r = mbedtls_rsa_import(rsa, &N, &P, &Q, NULL, &E); + if (r == 0) r = mbedtls_rsa_complete(rsa); + if (r == 0) r = mbedtls_rsa_check_privkey(rsa); + mbedtls_mpi_free(&P); + mbedtls_mpi_free(&Q); + mbedtls_mpi_free(&E); + mbedtls_mpi_free(&N); + mbedtls_mpi_free(&T); + mbedtls_mpi_free(&G); + return r; +} + /* OpenSSL's error for an RSA modulus length or public exponent it does not * generate, as Node reports it, or NULL. */ static const char *sp_rsa_keygen_failure(double bits, double exponent) { @@ -1773,16 +2967,22 @@ static SpKey *sp_keygen_run(const SpKeyGen *gen, bool *job_error) { scr_pki_throw_fmt(SCR_ERR_ERROR, NULL, "%s", failure); return NULL; } - /* mbedTLS generates even modulus lengths from 1024 to 8192 bits and, - * past the checks above, odd exponents up to INT_MAX. */ + /* Moduli past mbedTLS's 8192-bit numbers stay unsupported (OpenSSL + * generates up to 16384 bits). */ double b = gen->bits, e = gen->exponent; - if (b < MBEDTLS_RSA_GEN_KEY_MIN_BITS || b > MBEDTLS_MPI_MAX_BITS || fmod(b, 2) != 0 || e > 2147483647.0 || - mbedtls_pk_setup(&k->pk, mbedtls_pk_info_from_type(MBEDTLS_PK_RSA)) != 0 || - mbedtls_rsa_gen_key(mbedtls_pk_rsa(k->pk), sp_random, NULL, (unsigned)b, (int)e) != 0) { + if (b > MBEDTLS_MPI_MAX_BITS || mbedtls_pk_setup(&k->pk, mbedtls_pk_info_from_type(MBEDTLS_PK_RSA)) != 0 || + sp_rsa_generate(mbedtls_pk_rsa(k->pk), (unsigned)b, (uint64_t)e) != 0) { sp_key_release(k); sp_not_supported("generateKeyPairSync with this RSA modulus length or exponent"); return NULL; } + if (gen->pss) { + k->rsa_pss = true; + k->pss_params = gen->pss_params; + k->pss_md = gen->md; + k->pss_mgf1 = gen->mgf1; + k->pss_salt = gen->salt; + } return k; } if (mbedtls_pk_setup(&k->pk, mbedtls_pk_info_from_type(MBEDTLS_PK_ECKEY)) != 0 || @@ -1799,7 +2999,7 @@ static SpKey *sp_keygen_run(const SpKeyGen *gen, bool *job_error) { * pending, and *job_error as sp_keygen_run sets it. */ static ScrDyn *sp_key_pair(const ScrDyn *type, const ScrDyn *options, bool *job_error) { *job_error = false; - SpKeyGen gen; + SpKeyGen gen = {0}; if (!sp_keygen_check(type, options, &gen)) return NULL; SpKey *k = sp_keygen_run(&gen, job_error); if (!k) return NULL; @@ -1910,8 +3110,11 @@ static bool sp_algorithm(const ScrDyn *alg, mbedtls_md_type_t *md) { typedef struct { SpKey *key; /* +1 */ int padding; /* RSA_PKCS1_PADDING (1) or RSA_PKCS1_PSS_PADDING (6) */ - int salt; /* -1 digest length, -2 maximum (sign) or auto (verify) */ + int salt; /* -1 digest length, -2 maximum (sign) or auto (verify), + * -4 the digest length or the maximum, if smaller */ bool p1363; /* dsaEncoding: "ieee-p1363" */ + bool padding_given, salt_given; + mbedtls_md_type_t mgf1; /* PSS's MGF1 digest; NONE for the message's */ } SpSigKey; /* The key argument of a signing operation: a KeyObject, key material, or @@ -1932,6 +3135,9 @@ static bool sp_sig_key_plain(const ScrDyn *value, bool sign, SpSigKey *out) { out->padding = 1; out->salt = sign ? -2 : -2; out->p1363 = false; + out->padding_given = false; + out->salt_given = false; + out->mgf1 = MBEDTLS_MD_NONE; const ScrDyn *options = NULL; if (sp_is_object(value) && !sp_key_of(value) && !scr_buffer_storage_is(value)) { options = value; @@ -1948,6 +3154,7 @@ static bool sp_sig_key_plain(const ScrDyn *value, bool sign, SpSigKey *out) { return false; } out->padding = (int)padding->v.num; + out->padding_given = true; } const ScrDyn *salt = sp_opt(value, "saltLength"); if (!sp_absent(salt)) { @@ -1956,6 +3163,7 @@ static bool sp_sig_key_plain(const ScrDyn *value, bool sign, SpSigKey *out) { return false; } out->salt = (int)salt->v.num; + out->salt_given = true; } } SpKey *direct = sp_key_of(options ? sp_opt(options, "key") : value); @@ -2022,6 +3230,43 @@ static int sp_p1363_to_der(const unsigned char *sig, size_t len, unsigned char * static size_t sp_ec_size(SpKey *k) { return (mbedtls_pk_ec(k->pk)->MBEDTLS_PRIVATE(grp).nbits + 7) / 8; } +/* An RSA-PSS key's restrictions on a signature over a `md` digest, as + * OpenSSL's provider applies them: PSS padding only (the default), and + * for a key with RSASSA-PSS-params its digest, its MGF1 digest and salts + * of at least its own (the default). Without parameters a signature + * salts with the digest's length, or the most the key holds when that is + * less (RSA_PSS_SALTLEN_AUTO_DIGEST_MAX). False with OpenSSL's error + * pending. */ +static bool sp_pss_key_rules(SpSigKey *key, mbedtls_md_type_t md, size_t hash_len, bool sign) { + const SpKey *k = key->key; + if (!k->rsa_pss) return true; + if (key->padding_given && key->padding != 6) { + sp_throw_openssl("1C8000A5", "Provider routines", "illegal or unsupported padding mode", + "ERR_OSSL_ILLEGAL_OR_UNSUPPORTED_PADDING_MODE", true); + return false; + } + key->padding = 6; + if (!k->pss_params) { + if (!key->salt_given) key->salt = sign ? -4 : -2; + return true; + } + if (md != k->pss_md) { + sp_throw_openssl("1C8000AE", "Provider routines", "digest not allowed", "ERR_OSSL_DIGEST_NOT_ALLOWED", true); + return false; + } + key->mgf1 = k->pss_mgf1; + /* RSA_PSS_SALTLEN_AUTO (-2) is exempt from the key's minimum, as in + * OpenSSL: verification recovers the salt length from the signature and + * accepts any length (corpus 8178). */ + if (!key->salt_given) { + key->salt = k->pss_salt; + } else if (key->salt != -2 && (key->salt == -1 ? (int)hash_len : key->salt) < k->pss_salt) { + sp_throw_openssl("1C8000AC", "Provider routines", "pss saltlen too small", "ERR_OSSL_PSS_SALTLEN_TOO_SMALL", true); + return false; + } + return true; +} + /* Applies the RSA padding of a signature operation. */ static bool sp_rsa_padding(SpSigKey *key, mbedtls_md_type_t md) { if (!sp_is_rsa(key->key)) return true; @@ -2030,11 +3275,119 @@ static bool sp_rsa_padding(SpSigKey *key, mbedtls_md_type_t md) { return mbedtls_rsa_set_padding(rsa, MBEDTLS_RSA_PKCS_V15, MBEDTLS_MD_NONE) == 0; } +/* MGF1 over `mgf1` (RFC 8017 B.2.1), XORed into mask[0, len). */ +static int sp_mgf1_xor(mbedtls_md_type_t mgf1, const unsigned char *seed, size_t seed_len, unsigned char *mask, + size_t len) { + const mbedtls_md_info_t *info = mbedtls_md_info_from_type(mgf1); + size_t h = mbedtls_md_get_size(info); + unsigned char block[MBEDTLS_MD_MAX_SIZE]; + mbedtls_md_context_t ctx; + mbedtls_md_init(&ctx); + int r = mbedtls_md_setup(&ctx, info, 0); + for (uint32_t counter = 0; r == 0 && (size_t)counter * h < len; counter++) { + unsigned char c[4] = {(unsigned char)(counter >> 24), (unsigned char)(counter >> 16), + (unsigned char)(counter >> 8), (unsigned char)counter}; + if ((r = mbedtls_md_starts(&ctx)) == 0 && (r = mbedtls_md_update(&ctx, seed, seed_len)) == 0 && + (r = mbedtls_md_update(&ctx, c, 4)) == 0 && (r = mbedtls_md_finish(&ctx, block)) == 0) { + for (size_t i = 0; i < h && (size_t)counter * h + i < len; i++) mask[(size_t)counter * h + i] ^= block[i]; + } + } + mbedtls_md_free(&ctx); + mbedtls_platform_zeroize(block, sizeof block); + return r; +} + +/* An RSASSA-PSS signature whose MGF1 digest differs from the message's + * (RFC 8017 9.1.1 and 8.1.1), which mbedtls_rsa_rsassa_pss_sign_ext + * cannot make: the encoding here, then the private-key operation. */ +static int sp_pss_sign_mgf1(mbedtls_rsa_context *rsa, mbedtls_md_type_t md, mbedtls_md_type_t mgf1, + const unsigned char *hash, size_t hash_len, int salt_len, unsigned char *sig) { + size_t bytes = mbedtls_rsa_get_len(rsa); + size_t em_bits = mbedtls_rsa_get_bitlen(rsa) - 1; + size_t em_len = (em_bits + 7) / 8; + size_t salt = (size_t)salt_len; + if (em_len < hash_len + salt + 2) return MBEDTLS_ERR_RSA_BAD_INPUT_DATA; + unsigned char em[MBEDTLS_MPI_MAX_SIZE] = {0}; + unsigned char *db = em, *h = em + em_len - hash_len - 1; + size_t db_len = em_len - hash_len - 1; + unsigned char salt_bytes[MBEDTLS_MPI_MAX_SIZE]; + int r = sp_random(NULL, salt_bytes, salt); + /* H = Hash(0x00 * 8 || mHash || salt) */ + mbedtls_md_context_t ctx; + mbedtls_md_init(&ctx); + static const unsigned char zeros[8] = {0}; + if (r == 0 && (r = mbedtls_md_setup(&ctx, mbedtls_md_info_from_type(md), 0)) == 0 && + (r = mbedtls_md_starts(&ctx)) == 0 && (r = mbedtls_md_update(&ctx, zeros, 8)) == 0 && + (r = mbedtls_md_update(&ctx, hash, hash_len)) == 0 && (r = mbedtls_md_update(&ctx, salt_bytes, salt)) == 0) + r = mbedtls_md_finish(&ctx, h); + mbedtls_md_free(&ctx); + if (r == 0) { + /* DB = PS || 0x01 || salt, masked with MGF1(H). */ + db[db_len - salt - 1] = 0x01; + memcpy(db + db_len - salt, salt_bytes, salt); + r = sp_mgf1_xor(mgf1, h, hash_len, db, db_len); + } + if (r == 0) { + db[0] &= (unsigned char)(0xff >> (8 * em_len - em_bits)); + em[em_len - 1] = 0xbc; + unsigned char block[MBEDTLS_MPI_MAX_SIZE] = {0}; + memcpy(block + bytes - em_len, em, em_len); + r = mbedtls_rsa_private(rsa, sp_random, NULL, block, sig); + mbedtls_platform_zeroize(block, sizeof block); + } + mbedtls_platform_zeroize(em, sizeof em); + mbedtls_platform_zeroize(salt_bytes, sizeof salt_bytes); + return r; +} + +/* Verifies an RSASSA-PSS signature whose MGF1 digest differs from the + * message's (RFC 8017 9.1.2), which mbedtls_rsa_rsassa_pss_verify_ext + * cannot: it hashes M' with the MGF1 digest too. `salt_len` -2 accepts any + * salt length. */ +static bool sp_pss_verify_mgf1(mbedtls_rsa_context *rsa, mbedtls_md_type_t md, mbedtls_md_type_t mgf1, + const unsigned char *hash, size_t hash_len, int salt_len, + const unsigned char *sig, size_t sig_len) { + size_t bytes = mbedtls_rsa_get_len(rsa); + if (sig_len != bytes) return false; + unsigned char decoded[MBEDTLS_MPI_MAX_SIZE]; + if (mbedtls_rsa_public(rsa, sig, decoded) != 0) return false; + size_t em_bits = mbedtls_rsa_get_bitlen(rsa) - 1; + size_t em_len = (em_bits + 7) / 8; + if (bytes > em_len && decoded[0] != 0) return false; + unsigned char *em = decoded + (bytes - em_len); + if (em_len < hash_len + 2 || em[em_len - 1] != 0xbc) return false; + size_t db_len = em_len - hash_len - 1; + unsigned char *h = em + db_len; + unsigned char top = (unsigned char)(0xff >> (8 * em_len - em_bits)); + if (em[0] & (unsigned char)~top) return false; + unsigned char db[MBEDTLS_MPI_MAX_SIZE]; + memcpy(db, em, db_len); + if (sp_mgf1_xor(mgf1, h, hash_len, db, db_len) != 0) return false; + db[0] &= top; + size_t i = 0; + while (i < db_len && db[i] == 0) i++; + if (i == db_len || db[i] != 0x01) return false; + size_t salt = db_len - i - 1; + if (salt_len >= 0 && salt != (size_t)salt_len) return false; + static const unsigned char zeros[8] = {0}; + unsigned char check[MBEDTLS_MD_MAX_SIZE]; + mbedtls_md_context_t ctx; + mbedtls_md_init(&ctx); + int r; + if ((r = mbedtls_md_setup(&ctx, mbedtls_md_info_from_type(md), 0)) == 0 && (r = mbedtls_md_starts(&ctx)) == 0 && + (r = mbedtls_md_update(&ctx, zeros, 8)) == 0 && (r = mbedtls_md_update(&ctx, hash, hash_len)) == 0 && + (r = mbedtls_md_update(&ctx, db + i + 1, salt)) == 0) + r = mbedtls_md_finish(&ctx, check); + mbedtls_md_free(&ctx); + return r == 0 && memcmp(check, h, hash_len) == 0; +} + /* Signs a digest: +1 Buffer or NULL with the error pending. */ static ScrDyn *sp_sign_hash(SpSigKey *key, mbedtls_md_type_t md, const unsigned char *hash, size_t hash_len) { unsigned char sig[MBEDTLS_PK_SIGNATURE_MAX_SIZE]; size_t sig_len = 0; int r; + if (!sp_pss_key_rules(key, md, hash_len, true)) return NULL; if (!sp_rsa_padding(key, md)) { sp_unsupported_input(); return NULL; @@ -2047,12 +3400,15 @@ static ScrDyn *sp_sign_hash(SpSigKey *key, mbedtls_md_type_t md, const unsigned size_t max = bytes - hash_len - 2 - ((bits - 1) % 8 == 0 ? 1 : 0); if (salt == -1) salt = (int)hash_len; else if (salt == -2) salt = (int)max; + else if (salt == -4) salt = (int)(hash_len < max ? hash_len : max); if (salt < 0 || (size_t)salt > max) { sp_throw(SCR_ERR_ERROR, "ERR_OSSL_RSA_DATA_TOO_LARGE_FOR_KEY_SIZE", "error:1C8000A5:Provider routines::data too large for key size"); return NULL; } - r = mbedtls_rsa_rsassa_pss_sign_ext(rsa, sp_random, NULL, md, (unsigned)hash_len, hash, salt, sig); + r = key->mgf1 != MBEDTLS_MD_NONE && key->mgf1 != md + ? sp_pss_sign_mgf1(rsa, md, key->mgf1, hash, hash_len, salt, sig) + : mbedtls_rsa_rsassa_pss_sign_ext(rsa, sp_random, NULL, md, (unsigned)hash_len, hash, salt, sig); sig_len = bytes; } else { r = mbedtls_pk_sign(&key->key->pk, md, hash, hash_len, sig, sizeof sig, &sig_len, sp_random, NULL); @@ -2077,6 +3433,7 @@ static ScrDyn *sp_sign_hash(SpSigKey *key, mbedtls_md_type_t md, const unsigned * not verify or does not parse. */ static bool sp_verify_hash(SpSigKey *key, mbedtls_md_type_t md, const unsigned char *hash, size_t hash_len, const unsigned char *sig, size_t sig_len) { + if (!sp_pss_key_rules(key, md, hash_len, false)) return false; if (!sp_rsa_padding(key, md)) return false; if (sp_is_rsa(key->key) && key->padding == 6) { mbedtls_rsa_context *rsa = mbedtls_pk_rsa(key->key->pk); @@ -2084,6 +3441,9 @@ static bool sp_verify_hash(SpSigKey *key, mbedtls_md_type_t md, const unsigned c int salt = key->salt; if (salt == -1) salt = (int)hash_len; else if (salt == -2) salt = MBEDTLS_RSA_SALT_LEN_ANY; + if (key->mgf1 != MBEDTLS_MD_NONE && key->mgf1 != md) + return sp_pss_verify_mgf1(rsa, md, key->mgf1, hash, hash_len, salt == MBEDTLS_RSA_SALT_LEN_ANY ? -2 : salt, + sig, sig_len); return mbedtls_rsa_rsassa_pss_verify_ext(rsa, md, (unsigned)hash_len, hash, md, salt, sig) == 0; } if (sp_is_ec(key->key) && key->p1363) { @@ -2127,6 +3487,9 @@ ScrDyn *scr_crypto_sign_oneshot(ScrDyn *algorithm, ScrDyn *data, ScrDyn *key) { SpSigKey k; if (!sp_sig_key(key, true, &k)) return NULL; if (sp_reject_x25519(&k)) return NULL; + /* A null algorithm signs with an RSA-PSS key's own digest. */ + if ((algorithm->kind == SCR_DYN_UNDEF || algorithm->kind == SCR_DYN_NULL) && k.key->pss_params) + md = k.key->pss_md; ScrBytes *bytes = scr_bytes_buffer_source(data); if (sp_is_ed(k.key)) { ScrDyn *result = NULL; @@ -2165,6 +3528,8 @@ ScrDyn *scr_crypto_verify_oneshot(ScrDyn *algorithm, ScrDyn *data, ScrDyn *key, return NULL; } if (sp_reject_x25519(&k)) return NULL; + if ((algorithm->kind == SCR_DYN_UNDEF || algorithm->kind == SCR_DYN_NULL) && k.key->pss_params) + md = k.key->pss_md; ScrBytes *bytes = scr_bytes_buffer_source(data); ScrBytes *sig = scr_bytes_buffer_source(signature); if (sp_is_ed(k.key)) { @@ -2186,7 +3551,7 @@ ScrDyn *scr_crypto_verify_oneshot(ScrDyn *algorithm, ScrDyn *data, ScrDyn *key, scr_bytes_release(bytes); scr_bytes_release(sig); sp_key_release(k.key); - return scr_dyn_new_bool(ok); + return scr_exc_pending() ? NULL : scr_dyn_new_bool(ok); } /* ── Sign and Verify objects ──────────────────────────────────────────── */ @@ -2305,7 +3670,7 @@ static ScrDyn *sp_signer_invoke(void *h, ScrDyn *self, const char *method, ScrDy sig->data, (size_t)scr_bytes_byte_len(sig)); scr_bytes_release(sig); sp_key_release(k.key); - return scr_dyn_new_bool(ok); + return scr_exc_pending() ? NULL : scr_dyn_new_bool(ok); } scr_pki_throw_fmt(SCR_ERR_TYPE, NULL, "%s is not a function", what); return NULL; @@ -2466,7 +3831,11 @@ static const char sp_dh_public_types[] = /* Node's message names: a KeyObject's asymmetricKeyType. */ static const char *sp_type_name(const SpKey *k) { - return sp_is_ed(k) ? "ed25519" : sp_is_x(k) ? "x25519" : sp_is_rsa(k) ? "rsa" : "ec"; + return sp_is_ed(k) ? "ed25519" + : sp_is_x(k) ? "x25519" + : k->rsa_pss ? "rsa-pss" + : sp_is_rsa(k) ? "rsa" + : "ec"; } /* A key input of diffieHellman's options: a primitive other than a string @@ -2683,8 +4052,9 @@ ScrPkiKey *scr_pki_key_public_of(ScrPkiKey *key) { return sp_public_of(key); } ScrPkiKey *scr_pki_generate_rsa(unsigned bits, uint64_t exponent) { SpKey *k = sp_key_alloc(true); - if (exponent > 2147483647 || mbedtls_pk_setup(&k->pk, mbedtls_pk_info_from_type(MBEDTLS_PK_RSA)) != 0 || - mbedtls_rsa_gen_key(mbedtls_pk_rsa(k->pk), sp_random, NULL, bits, (int)exponent) != 0) { + if (bits < 512 || bits > MBEDTLS_MPI_MAX_BITS || exponent < 3 || exponent % 2 == 0 || + exponent > 4294967295u || mbedtls_pk_setup(&k->pk, mbedtls_pk_info_from_type(MBEDTLS_PK_RSA)) != 0 || + sp_rsa_generate(mbedtls_pk_rsa(k->pk), bits, exponent) != 0) { sp_key_release(k); return NULL; } diff --git a/packages/runtime/src/scr_rc.h b/packages/runtime/src/scr_rc.h index de9f21f44..171e420c2 100644 --- a/packages/runtime/src/scr_rc.h +++ b/packages/runtime/src/scr_rc.h @@ -703,8 +703,9 @@ void scr_cyc_set_growth_cap(long cap); /* Sets the nursery trigger SCR_CYCLE_THRESHOLD sets (scr_cycle.c): the * number of new candidates at which the release path runs a pass, for a * program that knows its own collection profile. The environment variable - * takes precedence; 0 restores the default. The trigger re-arms with it at - * the end of the next pass. Call it before the program allocates much. */ + * takes precedence; 0 restores the default, which starts at 256 and adapts + * to the program. The trigger re-arms with it at the end of the next pass. + * Call it before the program allocates much. */ void scr_cyc_set_nursery_threshold(long threshold); void scr_cyc_context_cleanup(void); diff --git a/packages/runtime/src/scr_regex.c b/packages/runtime/src/scr_regex.c index ab323c288..2a98e92c6 100644 --- a/packages/runtime/src/scr_regex.c +++ b/packages/runtime/src/scr_regex.c @@ -1416,6 +1416,51 @@ ScrStr *scr_regex_replace(ScrStr *s, ScrRegex *re, ScrStr *rep) { return scr_replace_impl(s, re, rep); } +void scr_re_replace_out_reserve(ScrReReplaceOut *o, int64_t need) { + if (need < 0 || (uint64_t)need > SCR_STR_MAX_CAP - (uint64_t)o->len) scr_regex_oom(); + size_t cap = (size_t)o->cap * 2; + if (cap < (size_t)(o->len + need)) cap = (size_t)(o->len + need); + if (cap > SCR_STR_MAX_CAP) cap = SCR_STR_MAX_CAP; + o->str->len = (size_t)o->len; + o->str = scr_str_regrow(o->str, cap); + o->data = o->str->data; + o->cap = (int64_t)o->str->cap; +} + +/* A global substitution with a literal ASCII replacement through the + * build-time replacer of its (pattern, replacement) pair: on an ASCII + * subject the replacer writes each gap and the replacement into the output + * and continues the search after every match without returning. Every + * other subject, and a plain-text pattern (string search), takes + * scr_replace_impl. */ +ScrStr *scr_regex_replace_with(ScrStr *s, ScrRegex *re, ScrStr *rep, + ScrRegexMatcherFn replacer) { + (void)scr_regex_bc(re); + if (scr_regex_lit(re) || !scr_text_ascii(s)) return scr_replace_impl(s, re, rep); + if (s->len > (size_t)INT32_MAX - 1) scr_regex_oom(); + re->last_index = 0; + ScrReReplaceOut o; + o.str = scr_str_alloc_raw(0, s->len + 64); + o.data = o.str->data; + o.len = 0; + o.cap = (int64_t)o.str->cap; + o.next = 0; + int32_t n = replacer((const uint8_t *)s->data, (int32_t)s->len, 0, (uint8_t **)(void *)&o); + if (n == 0) { + o.str->len = 0; + scr_str_release(o.str); + return scr_str_retain(s); + } + size_t tail = s->len - (size_t)o.next; + if ((int64_t)(tail + 1) > o.cap - o.len) scr_re_replace_out_reserve(&o, (int64_t)tail + 1); + scr_copy_bytes(o.data + o.len, s->data + o.next, tail); + o.len += (int64_t)tail; + o.str->len = (size_t)o.len; + o.data[o.len] = '\0'; + scr_str_note_ascii(o.str); + return o.str; +} + typedef struct { ScrDyn *arguments; int start, end; diff --git a/packages/runtime/src/scr_regex.h b/packages/runtime/src/scr_regex.h index cb61e49f3..502a29266 100644 --- a/packages/runtime/src/scr_regex.h +++ b/packages/runtime/src/scr_regex.h @@ -80,6 +80,22 @@ ScrStr *scr_regex_flags(ScrRegex *re); /* +1 */ ScrStr *scr_regex_to_string(ScrRegex *re); /* +1 */ /* replace: first match without /g, every match with it. */ ScrStr *scr_regex_replace(ScrStr *s, ScrRegex *re, ScrStr *rep); +/* The output of a build-time replacer (regex-matchers.ts): the replacer + * appends each gap and the replacement at data[len] after reserving room, + * and leaves in `next` the subject index the unmatched tail starts at. */ +typedef struct { + int64_t len; + int64_t cap; + int64_t next; + char *data; /* at offset 24 on every target */ + ScrStr *str; +} ScrReReplaceOut; +/* Grows o->str so that `need` more bytes fit at o->data[o->len]. */ +void scr_re_replace_out_reserve(ScrReReplaceOut *o, int64_t need); +/* A global substitution with a literal ASCII replacement, through the + * replacer built for that pattern and replacement. Borrows all. */ +ScrStr *scr_regex_replace_with(ScrStr *s, ScrRegex *re, ScrStr *rep, + ScrRegexMatcherFn replacer); /* replace/replaceAll with a replacer closure of `arity` (at most * SCR_REPLACER_MAX_ARITY) parameters. Parameter i receives what Node passes * at position i: the match and captures below the capture count n, the diff --git a/packages/runtime/src/scr_string.c b/packages/runtime/src/scr_string.c index 1e7802fbb..64a4772e5 100644 --- a/packages/runtime/src/scr_string.c +++ b/packages/runtime/src/scr_string.c @@ -3514,6 +3514,8 @@ ScrStr *scr_bool_to_scrstr(bool b) { * decode is a whole char; an i addressing the LOW half of an astral pair * (unreachable from the iterator, which only lands on char starts) answers * the containing char. Out of range → empty string. Borrows s; +1. */ +static ScrStr *scr_str_char_of(const char *bytes, size_t adv); + ScrStr *scr_str_cp_at(ScrStr *s, double i) { double idx = scr_to_integer_or_infinity(i); if (!(idx >= 0)) return scr_str_empty(); @@ -3528,7 +3530,80 @@ ScrStr *scr_str_cp_at(ScrStr *s, double i) { if (off >= s->len) return scr_str_empty(); /* out of range */ size_t adv; (void)scr_utf8_decode(s->data + off, &adv); - return scr_str_from_span(s->data + off, adv); + return scr_str_char_of(s->data + off, adv); +} + +/* The one-character strings of U+0080..U+07FF (two-byte sequences, lead + * bytes 0xC2..0xDF), made immortal on first use and shared by every + * thread, as the ASCII ones are static: the string iterator returns them + * instead of a new string per character. A string is immortal before its + * slot publishes it; a thread that loses the race to fill a slot keeps its + * own immortal copy, so at most one extra string per slot and thread. The + * strings come from the system allocator, as other state that outlives a + * thread does: a worker's own memory goes back when it exits without + * sharing (scr_alloc.c), and the table outlives the worker that filled a + * slot. */ +static ScrStr *scr_two_byte_chars[0x800 - 0x80]; + +static ScrStr *scr_two_byte_char(const char *bytes) { + uint32_t cp = ((uint32_t)((unsigned char)bytes[0] & 0x1F) << 6) | ((unsigned char)bytes[1] & 0x3F); + ScrStr **slot = &scr_two_byte_chars[cp - 0x80]; + ScrStr *known = __atomic_load_n(slot, __ATOMIC_ACQUIRE); + if (known) return known; + ScrStr *made = calloc(1, sizeof(ScrStr) + 3); + if (!made) scr_oom(); + made->len = 2; + made->cap = 2; + memcpy(made->data, bytes, 2); + scr_str_hash_prime(made); + made->rc = SIZE_MAX; + scr_lsan_immortal(made); + ScrStr *expected = NULL; + if (__atomic_compare_exchange_n(slot, &expected, made, false, __ATOMIC_ACQ_REL, + __ATOMIC_ACQUIRE)) + return made; + return expected; +} + +/* The character whose encoding starts at bytes and is adv bytes long (+1, + * or immortal). */ +static ScrStr *scr_str_char_of(const char *bytes, size_t adv) { + unsigned char lead = (unsigned char)bytes[0]; + if (adv == 1 && lead < 0x80) return (ScrStr *)&scr_ascii1[lead]; + if (adv == 2) return scr_two_byte_char(bytes); + return scr_str_from_span(bytes, adv); +} + +ScrStr *scr_str_cp_at_byte(ScrStr *s, uint64_t b) { + if (b >= s->len) return scr_str_empty(); + size_t adv; + (void)scr_utf8_decode(s->data + b, &adv); + return scr_str_char_of(s->data + b, adv); +} + +ScrArr *scr_str_code_points(ScrStr *s) { + /* Every byte but a continuation byte starts one character; a lone + * surrogate's three bytes are one character. */ + size_t count = s->len; + if (!s->ascii) + for (size_t b = 0; b < s->len; b++) count -= ((unsigned char)s->data[b] & 0xC0) == 0x80; + ScrArr *out = scr_arr_new_inline(SCR_ELEM_STR, count, false); + size_t k = 0; + for (size_t b = 0; b < s->len;) { + unsigned char c = (unsigned char)s->data[b]; + if (c < 0x80) { + out->data[k++] = (uint64_t)(uintptr_t)&scr_ascii1[c]; + b++; + continue; + } + size_t adv; + (void)scr_utf8_decode(s->data + b, &adv); + out->data[k++] = (uint64_t)(uintptr_t)scr_str_char_of(s->data + b, adv); + b += adv; + } + out->len = (uint32_t)count; + out->filled = (uint32_t)count; + return out; } /* ── the URI component codecs (encodeURIComponent/decodeURIComponent) ── diff --git a/packages/runtime/src/scr_string_methods.h b/packages/runtime/src/scr_string_methods.h index 01ccdd8ae..2e69ee6ac 100644 --- a/packages/runtime/src/scr_string_methods.h +++ b/packages/runtime/src/scr_string_methods.h @@ -150,6 +150,15 @@ ScrStr *scr_str_char_at(ScrStr *s, double i); * (two units); the consuming loop advances by the result's length. * Out of range → empty string. Returns +1. */ ScrStr *scr_str_cp_at(ScrStr *s, double i); +/* The string iterator's step over the WTF-8 bytes: the code-point + * character whose encoding starts at byte offset b, a character start + * below s->len (a lone surrogate is its own three-byte character). The + * compiler answers ASCII bytes inline and calls this for the rest. + * Returns +1. */ +ScrStr *scr_str_cp_at_byte(ScrStr *s, uint64_t b); +/* [...s] and Array.from(s): the code-point characters of s in order, as a + * fresh string array of exactly that length. Returns +1. */ +struct ScrArr *scr_str_code_points(ScrStr *s); /* trimStart()/trimEnd(): the one-sided halves of trim. Returns +1. */ ScrStr *scr_str_trim_start(ScrStr *s); diff --git a/packages/runtime/src/scr_webcrypto_pki.c b/packages/runtime/src/scr_webcrypto_pki.c index 7439a12db..458cfeae8 100644 --- a/packages/runtime/src/scr_webcrypto_pki.c +++ b/packages/runtime/src/scr_webcrypto_pki.c @@ -380,7 +380,8 @@ static ScrPromise *sw_generate_key(const ScrDyn *args) { scr_throw_domex("OperationError", "algorithm.publicExponent must be odd"); goto done; } - if (bits > 16384 || value < 3) { + /* The pki unit's numbers hold moduli up to 8192 bits. */ + if (bits > 8192 || value < 3) { scr_wc_unsupported("generateKey", "this RSA modulus length or public exponent"); goto done; } diff --git a/packages/runtime/test/test_array.c b/packages/runtime/test/test_array.c index 7ca0ebd1a..9126c154f 100644 --- a/packages/runtime/test/test_array.c +++ b/packages/runtime/test/test_array.c @@ -2513,6 +2513,84 @@ static void test_states_on_demand(void) { #endif } +/* Elements outside the element type (SCR_ARR_FOREIGN): stores, reads, + * copies, sorting and removal keep the stored value's count balanced. */ +static void test_foreign_elements(void) { + ScrStr *text = scr_str_new("x", 1); + ScrDyn *x = scr_dyn_new_str(text); + scr_str_release(text); + ScrArr *a = scr_arr_new(SCR_ELEM_F64, 0); + scr_arr_push_f64(a, 1); + scr_arr_push_f64(a, 2); + check(scr_arr_set_foreign(a, 2, x), "foreign store appends"); + check_f64(scr_arr_len(a), 3, "foreign store extends the length"); + check_f64(scr_arr_state(a, 2), SCR_ARR_FOREIGN, "foreign element state"); + ScrDyn *got = scr_arr_get_foreign(a, 2); + check(got == x, "get_foreign answers the stored value"); + scr_dyn_release(got); + check(scr_arr_get_foreign(a, 0) == NULL, "a number is not a foreign element"); + check(isnan(scr_arr_get_number(a, 2)), "numeric read of a foreign 'x' is NaN"); + check(scr_arr_set_foreign(a, 3, scr_dyn_undefined()), "undefined store"); + check_f64(scr_arr_state(a, 3), SCR_ARR_UNDEFINED, "undefined is a present undefined"); + ScrStr *sep = scr_str_new(",", 1); + ScrStr *joined = scr_arr_join(a, sep); + check(joined->len == 6 && memcmp(joined->data, "1,2,x,", 6) == 0, "join converts foreign"); + scr_str_release(joined); + ScrArr *copy = scr_arr_slice(a, 0, INFINITY); + check_f64(scr_arr_state(copy, 2), SCR_ARR_FOREIGN, "slice keeps a foreign element"); + scr_arr_release(copy); + ScrArr *head = scr_arr_new(SCR_ELEM_F64, 0); + scr_arr_set_foreign(head, 0, x); + scr_arr_push_f64(head, 2); + scr_arr_push_f64(head, 3); + copy = scr_arr_slice(head, 1, INFINITY); + check_f64(scr_arr_get_f64(copy, 1), 3, "slice past a foreign element"); + scr_arr_release(copy); + scr_arr_release(head); + scr_arr_set_f64(a, 2, 5); + check_f64(scr_arr_get_f64(a, 2), 5, "a number replaces a foreign element"); + scr_arr_set_foreign(a, 2, x); + scr_arr_set_len(a, 1); + check_f64(scr_arr_len(a), 1, "truncation drops foreign elements"); + scr_arr_set_foreign(a, 1, x); + check(scr_arr_delete(a, 1), "delete removes a foreign element"); + scr_arr_set_foreign(a, 1, x); + uint64_t slot = 0; + check(scr_arr_pop_state(a, &slot) == SCR_ARR_FOREIGN, "pop answers the foreign state"); + check((ScrDyn *)(uintptr_t)slot == x, "pop moves the value out"); + scr_dyn_release((ScrDyn *)(uintptr_t)slot); + for (int i = 0; i < 12; i++) scr_arr_set_foreign(a, (double)(1 + i), x); + double seven = 7; + uint64_t inserted[1]; + memcpy(&inserted[0], &seven, sizeof seven); + scr_arr_splice_drop(a, 1, 10, 1, inserted); + check_f64(scr_arr_len(a), 4, "splice removes foreign elements"); + check_f64(scr_arr_get_f64(a, 1), 7, "splice inserts after them"); + scr_arr_release(a); + + ScrStr *letter = scr_str_new("a", 1); + ScrDyn *a_value = scr_dyn_new_str(letter); + scr_str_release(letter); + ScrArr *s = scr_arr_new(SCR_ELEM_F64, 0); + scr_arr_push_f64(s, 10); + scr_arr_push_f64(s, 9); + scr_arr_set_foreign(s, 2, a_value); + scr_arr_push_f64(s, 1); + scr_arr_release(scr_arr_sort_primitive(s, false)); + check(scr_arr_get_f64(s, 0) == 1 && scr_arr_get_f64(s, 1) == 10 && scr_arr_get_f64(s, 2) == 9, + "default sort orders numbers by their strings"); + check_f64(scr_arr_state(s, 3), SCR_ARR_FOREIGN, "default sort orders 'a' after the digits"); + scr_arr_release(s); + scr_dyn_release(a_value); + + ScrArr *flags = scr_arr_new(SCR_ELEM_BOOL, 0); + check(!scr_arr_set_foreign(flags, 0, x), "boolean arrays hold no foreign elements"); + scr_arr_release(flags); + scr_str_release(sep); + check(x->rc == 1, "every foreign store released its count"); + scr_dyn_release(x); +} + int main(int argc, char **argv) { if (argc > 1) { ScrArr *a = scr_arr_new(SCR_ELEM_F64, 0); @@ -2573,6 +2651,8 @@ int main(int argc, char **argv) { test_head_room(); test_states_on_demand(); + test_foreign_elements(); + fprintf(stderr, "%ld/%ld cases passed\n", total - failed, total); return failed == 0 ? 0 : 1; } diff --git a/packages/runtime/test/test_bytes.c b/packages/runtime/test/test_bytes.c index 8c6602948..1aea7a2d0 100644 --- a/packages/runtime/test/test_bytes.c +++ b/packages/runtime/test/test_bytes.c @@ -216,6 +216,135 @@ static void test_encodings(void) { scr_str_release(b64); } +/* The length of the well-formed WTF-8 sequence at in[i], or 0. */ +static size_t ref_seq_len(const uint8_t *in, size_t n, size_t i) { + uint8_t b = in[i]; + if (b < 0x80) return 1; + size_t len; + uint8_t lo = 0x80, hi = 0xbf; + if (b >= 0xc2 && b <= 0xdf) len = 2; + else if (b >= 0xe0 && b <= 0xef) len = 3; + else if (b >= 0xf0 && b <= 0xf4) len = 4; + else return 0; + if (b == 0xe0) lo = 0xa0; + if (b == 0xf0) lo = 0x90; + if (b == 0xf4) hi = 0x8f; + if (n - i < len || in[i + 1] < lo || in[i + 1] > hi) return 0; + for (size_t k = 2; k < len; k++) + if ((in[i + k] & 0xc0) != 0x80) return 0; + return len; +} + +/* The UTF-16 units of string storage, one at a time: a well-formed + * sequence gives its code point's units and any other byte stands for + * U+0000..U+00FF (a raw HTTP header byte). */ +static size_t ref_units(const uint8_t *in, size_t n, uint16_t *out) { + size_t k = 0; + for (size_t i = 0; i < n;) { + size_t len = ref_seq_len(in, n, i); + if (len == 0) { + out[k++] = in[i++]; + continue; + } + uint32_t cp = len == 1 ? in[i] : (uint32_t)(in[i] & (0x7f >> len)); + for (size_t j = 1; j < len; j++) cp = (cp << 6) | (in[i + j] & 0x3f); + i += len; + if (cp > 0xffff) { + out[k++] = (uint16_t)(0xd800 + ((cp - 0x10000) >> 10)); + out[k++] = (uint16_t)(0xdc00 + ((cp - 0x10000) & 0x3ff)); + } else { + out[k++] = (uint16_t)cp; + } + } + return k; +} + +/* Encodes storage that is not well-formed (raw header bytes become string + * storage unchecked) and compares latin1, ascii and utf16le with the + * reference; hex and base64 decode the latin1 bytes. ASan checks that no + * path reads or writes outside either buffer. */ +/* encs: latin1, ascii, utf16le, hex, base64. */ +static void check_raw_encodings(ScrStr *const *encs, const uint8_t *in, size_t n, const char *what) { + ScrStr *latin1 = encs[0], *ascii = encs[1], *utf16le = encs[2], *hex = encs[3], *b64 = encs[4]; + ScrStr *s = scr_str_new((const char *)in, n); + uint16_t units[64]; + size_t k = ref_units(in, n, units); + ScrStr *narrow[] = {latin1, ascii}; + for (size_t e = 0; e < 2; e++) { + ScrBytes *b = scr_bytes_from_str(s, narrow[e]); + bool ok = b->len == k; + for (size_t j = 0; ok && j < k; j++) ok = b->data[j] == (uint8_t)units[j]; + check(ok, what); + scr_bytes_release(b); + } + ScrBytes *wide = scr_bytes_from_str(s, utf16le); + bool ok = wide->len == 2 * k; + for (size_t j = 0; ok && j < k; j++) + ok = wide->data[2 * j] == (uint8_t)units[j] && wide->data[2 * j + 1] == units[j] >> 8; + check(ok, what); + scr_bytes_release(wide); + scr_bytes_release(scr_bytes_from_str(s, hex)); + scr_bytes_release(scr_bytes_from_str(s, b64)); + scr_str_release(s); +} + +static void test_raw_storage_encodings(void) { + /* Each pattern at every offset in an eight-byte word, alone and followed + * by ASCII, two-byte characters or more of the pattern. */ + static const struct { const char *bytes; size_t n; } patterns[] = { + {"\x80", 1}, {"\x80\x81\xbf\x90", 4}, {"\xc3", 1}, {"\xc3\xa9", 2}, + {"\xc3\xa9\xc3", 3}, {"\xc0\x80\xc1\xbf", 4}, {"\xe9\x41\xe9", 3}, + {"\xf0\x9f\x41", 3}, {"\xe2\x82", 2}, {"\xe2\x82\xac", 3}, + {"\xf0\x9f\x98\x80", 4}, {"\xed\xa0\x80", 3}, {"\xff\xfe\xf5", 3}, + {"\xdf\xbf\x80\x80", 4}, {"\xc2\x80\xc2", 3}, {"x\xe9y\x80\x80z\xdf", 7}, + }; + static const struct { const char *bytes; size_t n; } tails[] = { + {"", 0}, {"bcdefghij", 9}, {"\xc3\xa9\xc3\xa9\xc3\xa9\xc3\xa9", 8}, {"\x80\xc3\xa9", 3}, + }; + ScrStr *encs[] = {S("latin1"), S("ascii"), S("utf16le"), S("hex"), S("base64")}; + uint8_t buf[64]; + char what[96]; + for (size_t p = 0; p < sizeof patterns / sizeof patterns[0]; p++) { + for (size_t offset = 0; offset < 17; offset++) { + for (size_t t = 0; t < sizeof tails / sizeof tails[0]; t++) { + for (size_t lead = 0; lead < 2; lead++) { + /* lead: the offset is two-byte characters instead of ASCII. */ + size_t n = 0; + for (size_t j = 0; j < offset; j++) { + if (lead && j + 1 < offset) { + buf[n++] = 0xc3; + buf[n++] = 0xa9; + j++; + } else { + buf[n++] = 'a'; + } + } + memcpy(buf + n, patterns[p].bytes, patterns[p].n); + n += patterns[p].n; + memcpy(buf + n, tails[t].bytes, tails[t].n); + n += tails[t].n; + snprintf(what, sizeof what, "raw storage encodings: pattern %zu offset %zu tail %zu lead %zu", + p, offset, t, lead); + check_raw_encodings(encs, buf, n, what); + } + } + } + } + /* Random bytes, mostly high, of every length up to 40. */ + uint32_t seed = 12345; + for (int round = 0; round < 4000; round++) { + size_t n = (size_t)round % 41; + for (size_t j = 0; j < n; j++) { + seed = seed * 1103515245u + 12345u; + uint8_t r = (uint8_t)(seed >> 16); + buf[j] = (seed >> 27) & 1 ? (uint8_t)(0x80 | r) : (uint8_t)(r & 0x7f); + } + snprintf(what, sizeof what, "raw storage encodings: random round %d", round); + check_raw_encodings(encs, buf, n, what); + } + for (size_t e = 0; e < sizeof encs / sizeof encs[0]; e++) scr_str_release(encs[e]); +} + static void test_concat_and_u32be(void) { ScrArr *list = scr_arr_new(SCR_ELEM_BYTES, 0); ScrBytes *a = scr_bytes_new(SCR_BYTES_U8, 2); @@ -445,6 +574,7 @@ int main(int argc, char **argv) { scr_bytes_test_bulk_storage(); scr_bytes_test_buffer_pool(); test_encodings(); + test_raw_storage_encodings(); test_concat_and_u32be(); test_from_arr_and_random(); test_zlib_roundtrip(); diff --git a/packages/runtime/test/test_cycle.c b/packages/runtime/test/test_cycle.c index 0ab827e47..dc48abd26 100644 --- a/packages/runtime/test/test_cycle.c +++ b/packages/runtime/test/test_cycle.c @@ -575,6 +575,60 @@ static void check_program_nursery_threshold(void) { scr_collect_cycles(); } +/* How many dead rings the release path takes to run its next pass, from + * the end of the previous one: the nursery trigger in effect. */ +static size_t nursery_window(void) { + size_t before = freed, n = 0; + while (freed == before && n < 65536) { + release_live(make_ring(1)); + n++; + } + check(freed - before == n, "a nursery pass left dead rings behind"); + return n; +} + +/* A scheduled full pass that reclaims nothing: its only candidate is an + * older live leaf. */ +static void scheduled_pass_finding_nothing(Node *leaf) { + size_t before = freed; + read_live(leaf); + for (size_t i = 0; i < configured_nursery_threshold(); i++) + scr_cyc_collect_scheduled(); + check(freed == before, "a scheduled pass over a live leaf freed something"); +} + +/* Unpinned, the nursery trigger starts at 256 candidates. Each scheduled + * full pass that reclaims promoted rings doubles it, up to 4096; each one + * that reclaims nothing after the nursery passes freed rings halves it, + * down to 256. */ +static void check_adaptive_nursery_threshold(void) { + if (getenv("SCR_CYCLE_THRESHOLD") != NULL) return; /* it pins the trigger */ + scr_cyc_set_nursery_threshold(0); + scr_collect_cycles(); + check(nursery_window() == 256, "the unpinned nursery trigger did not start at 256"); + static const size_t widened[] = { 512, 1024, 2048, 4096, 4096 }; + for (size_t i = 0; i < sizeof(widened) / sizeof(widened[0]); i++) { + collect_dead_mature_rings(); + check(nursery_window() == widened[i], + "a full pass reclaiming promoted rings did not double the nursery trigger"); + } + + static const size_t narrowed[] = { 2048, 1024, 512, 256, 256 }; + Node *leaf = make_leaf(); + read_live(leaf); + scr_collect_cycles(); /* promote */ + for (size_t i = 0; i < sizeof(narrowed) / sizeof(narrowed[0]); i++) { + scheduled_pass_finding_nothing(leaf); + check(nursery_window() == narrowed[i], + "a full pass reclaiming nothing did not halve the nursery trigger"); + } + leaf->rc--; + scr_cyc_on_dead(leaf); + node_free(leaf); + scr_cyc_set_nursery_threshold(0); + scr_collect_cycles(); +} + /* The phase sweep (scr_collect_cycles_phase) is for a caller that knows a * phase's data has just died: it sweeps, and the growth schedule starts * over. After passes that find nothing have widened the old level's @@ -1506,6 +1560,9 @@ static void check_acyclic_objects(void) { } int main(void) { + /* The scheduling checks count candidates against a fixed nursery trigger; + * check_adaptive_nursery_threshold covers the unpinned one. */ + scr_cyc_set_nursery_threshold((long)configured_nursery_threshold()); check_inline_buffering(); check_old_garbage_reclaimed_while_growing(); check_older_rings_into_old(); @@ -1541,6 +1598,7 @@ int main(void) { check_acyclic_objects(); check_program_growth_cap(); check_program_nursery_threshold(); + check_adaptive_nursery_threshold(); printf("cycle collection checks passed: threshold=%zu growth_cap=%zu\n", configured_nursery_threshold(), configured_growth_cap()); return 0; diff --git a/packages/threads/package.json b/packages/threads/package.json index c29967cc6..6ebcf5b8b 100644 --- a/packages/threads/package.json +++ b/packages/threads/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/threads", - "version": "0.4.4", + "version": "0.4.5", "description": "Zero-copy publication of immutable object graphs to worker threads (scriptc), with a deep-freezing Node.js implementation", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/packages/types/package.json b/packages/types/package.json index db88b4b2f..b2aa805be 100644 --- a/packages/types/package.json +++ b/packages/types/package.json @@ -1,6 +1,6 @@ { "name": "@scriptc/types", - "version": "0.4.4", + "version": "0.4.5", "description": "Contract types for scriptc: type-only annotations (Int32, NoThrow, Immutable) that scriptc checks and optimizes, with no runtime semantics", "license": "Apache-2.0", "homepage": "https://scriptc.dev", diff --git a/tests/corpus-node-islands-order/p12-island-io-vs-busy/main.ts b/tests/corpus-node-islands-order/p12-island-io-vs-busy/main.ts index 7b342dd4d..477175473 100644 --- a/tests/corpus-node-islands-order/p12-island-io-vs-busy/main.ts +++ b/tests/corpus-node-islands-order/p12-island-io-vs-busy/main.ts @@ -1,4 +1,5 @@ // @requires-libnode +// @order: race // X { R { L i-io } } B 30 L h-sync T 0 { L h-t0 } import { callSync } from "@scriptc/node"; import { readFile, writeSync } from "node:fs"; diff --git a/tests/corpus/1561-forof-strings.ts b/tests/corpus/1561-forof-strings.ts index 9d16efd9a..0ea09e755 100644 --- a/tests/corpus/1561-forof-strings.ts +++ b/tests/corpus/1561-forof-strings.ts @@ -1,7 +1,7 @@ // for-of over STRINGS: JS's string iterator walks code POINTS — each pass // binds the whole character, so astral chars arrive as one two-unit string -// (where charAt/slice would truncate the halves). Desugars over the UTF-16 -// cursor with the cpAt step intrinsic. +// (where charAt/slice would truncate the halves). Desugars over a byte +// cursor with the cpAtByte step intrinsic. for (const ch of "héllo 🎉x") console.log(ch, ch.length); // The empty string iterates zero times. diff --git a/tests/corpus/6067-view-holes-in-typed-arrays.ts b/tests/corpus/6067-view-holes-in-typed-arrays.ts new file mode 100644 index 000000000..831b3be7a --- /dev/null +++ b/tests/corpus/6067-view-holes-in-typed-arrays.ts @@ -0,0 +1,115 @@ +// Holes written through an `unknown[]` or `any` view of a typed array are +// holes of the typed array: for...of visits them as undefined and the find +// family passes them to its callback, as in Node. The views reach the array +// through a binding, an `any` parameter, a generic instantiated with +// `unknown`, a record field, a class field and a nested array. + +const numbers: number[] = [1, 2]; +const numbersView: unknown[] = numbers; +numbersView[3] = 5; +for (const value of numbers) console.log("numbers", value, typeof value); +console.log(numbers.findIndex((value) => value === undefined)); +console.log(numbers.findLastIndex((value) => value === undefined)); +console.log(numbers.find((value) => value !== undefined && value > 1)); + +const grown: string[] = ["a"]; +const grownView: unknown[] = grown; +grownView.length = 3; +for (const value of grown) console.log("grown", value); + +function growAny(values: any): void { + values[2] = 9; +} +const passed: number[] = [7]; +growAny(passed); +for (const value of passed) console.log("passed", value); + +function keep(value: T): T { + return value; +} +const kept: number[] = [3]; +(keep(kept) as unknown[])[2] = 4; +for (const value of kept) console.log("kept", value); + +const boxed: number[] = [1]; +const box: unknown = { values: boxed }; +(box as { values: unknown[] }).values[2] = 3; +for (const value of boxed) console.log("boxed", value); + +class Holder { + values: number[] = [1, 2]; +} +const holder = new Holder(); +const holderView: unknown = holder; +(holderView as { values: unknown[] }).values[3] = 4; +for (const value of holder.values) console.log("holder", value); + +const nested: number[][] = [[1]]; +const nestedView: unknown[][] = nested; +nestedView[0]![2] = 3; +for (const value of nested[0]!) console.log("nested", value); + +// A named property written through a view is not an element. +const named: number[] = [1, 2]; +const bag = named as unknown as Record; +bag["label"] = "x"; +for (const value of named) console.log("named", value); +console.log(named.length, bag["label"]); + +// Views that reach dynamic code through class field initializers, +// parameter defaults, logical assignments, yield and throw, and through +// standard library calls that keep their argument or pass it to a +// callback: Promise.resolve, push onto an `unknown[]`, Map.prototype.set +// and a JSON.stringify replacer. +const fieldNumbers: number[] = [1, 2]; +class FieldView { + view: unknown[] = fieldNumbers; +} +new FieldView().view[3] = 4; +for (const value of fieldNumbers) console.log("field", value); +const defaultNumbers: number[] = [1]; +function extend(view: unknown[] = defaultNumbers): void { + view.length = 3; +} +extend(); +console.log(defaultNumbers.findIndex((value) => value === undefined), defaultNumbers.length); +let assigned: unknown[] | undefined; +const assignedNumbers: number[] = [5]; +assigned ??= assignedNumbers; +assigned[2] = 6; +for (const value of assignedNumbers) console.log("assigned", value); +const yieldedNumbers: number[] = [7]; +function* yieldViews(): Generator { + yield yieldedNumbers; +} +for (const view of yieldViews()) view[2] = 8; +console.log(yieldedNumbers.findLastIndex((value) => value === undefined), yieldedNumbers.length); +const thrownNumbers: number[] = [9]; +try { + throw thrownNumbers; +} catch (error) { + (error as unknown[])[2] = 10; +} +for (const value of thrownNumbers) console.log("thrown", value); +const retainedNumbers: number[] = [12]; +const retained: unknown[] = []; +retained.push(retainedNumbers); +(retained[0] as unknown[])[2] = 13; +for (const value of retainedNumbers) console.log("kept", value); +const mappedNumbers: number[] = [14]; +const byName = new Map(); +byName.set("a", mappedNumbers); +(byName.get("a") as unknown[])[2] = 15; +console.log(mappedNumbers.findIndex((value) => value === undefined), mappedNumbers.length); +const replacedNumbers: number[] = [16]; +JSON.stringify({ replacedNumbers: replacedNumbers }, (key, value: unknown) => { + if (key === "replacedNumbers") (value as unknown[])[2] = 17; + return value; +}); +for (const value of replacedNumbers) console.log("replaced", value); +const resolvedNumbers: number[] = [11]; +void Promise.resolve(resolvedNumbers).then((view) => { + (view as unknown[])[2] = 18; + for (const value of resolvedNumbers) console.log("resolved", value); +}); + diff --git a/tests/corpus/6068-foreign-elements-through-views.ts b/tests/corpus/6068-foreign-elements-through-views.ts new file mode 100644 index 000000000..1b23f6fa7 --- /dev/null +++ b/tests/corpus/6068-foreign-elements-through-views.ts @@ -0,0 +1,202 @@ +// A number[] or string[] seen as `unknown[]` holds the values written +// through the view that its element type cannot hold, as in Node. The view +// and the typed array are the same array: length, the view's reads and +// iteration, printing, JSON, join, searches, copies, Math over a spread, +// sorting and truncation see the written values. + +const numbers: number[] = [1, 2]; +const numbersView: unknown[] = numbers; +numbersView.push("x"); +numbersView[0] = true; +console.log(numbers.length, numbersView.length, numbers === numbersView); +console.log(numbersView[0], numbersView[2], typeof numbersView[2]); +console.log(numbers, String(numbers), numbers.join("|"), JSON.stringify(numbers)); +for (const value of numbersView) console.log("value", value, typeof value); +console.log(numbers.indexOf(2), numbers.includes(2)); +console.log([...numbers], numbers.slice(1), numbers.slice(1, 2), numbers.concat([9])); + +// Math over a spread applies ToNumber to every element, as in Node. +const digits: number[] = [5]; +const digitsView: unknown[] = digits; +digitsView.push("4", null); +console.log(Math.max(...digits), Math.min(...digits), Math.hypot(...digits)); + +// Present undefined, null and objects keep their own values. +const mixed: number[] = [1]; +const mixedView: unknown[] = mixed; +mixedView.push(undefined, null, { a: 1 }, [2, 3]); +console.log(mixed, mixed.length, 1 in mixed, mixedView[2] === null); +console.log(JSON.stringify(mixed), mixed.join("-")); + +// Writing a number over a foreign element makes it a number again. +const replaced: number[] = [1, 2]; +const replacedView: unknown[] = replaced; +replacedView.push("x"); +replaced[2] = 5; +console.log(replaced, replacedView, replaced[2] + 1); + +// Default sort orders every element by its string, as in Node. +const sorted: number[] = [10, 9, 1]; +const sortedView: unknown[] = sorted; +sortedView.push("b", "a"); +console.log(sorted.sort(), sorted.length, sortedView[4]); + +// Truncation drops written values; growing again leaves holes. +const truncated: number[] = [1]; +const truncatedView: unknown[] = truncated; +truncatedView.push("x", "y"); +truncated.length = 1; +truncated.length = 3; +console.log(truncated, truncatedView.length); + +// string[] holds values of other types the same way. +const words: string[] = ["a"]; +const wordsView: unknown[] = words; +wordsView.push(1, false); +console.log(words, words.join("-"), words.length, typeof wordsView[1]); + +// A conversion that changes the array does not change the spread's values, +// and toSorted reads holes as undefined. +const grown: number[] = [3, 1]; +const grownView: unknown[] = grown; +grownView.push({ + valueOf: () => { + for (let i = 0; i < 64; i++) grown.push(i); + grown.length = 1; + return 2; + }, +}); +console.log(Math.max(...grown), Math.min(...grown), grown.length); +const holey: string[] = ["c", "a"]; +const holeyView: unknown[] = holey; +holeyView[4] = 7; +holeyView[6] = undefined; +console.log(holey.toSorted(), holey.length); +holey.sort(); +console.log(holey, holey.length); + +// Conversions during an in-place sort that change the array's length: +// elements added past the sorted length stay. +const appended: number[] = [3, 1, 2]; +const appendedView: unknown[] = appended; +let appendedCalls = 0; +appendedView[1] = { + toString(): string { + if (appendedCalls++ === 0) appended.push(7, 8); + return "1"; + }, +}; +appended.sort(); +console.log(appended.length, appended.slice(3), String(appendedView[0]), appended[2]); +const reduced: number[] = [3, 1, 2]; +const reducedView: unknown[] = reduced; +reducedView[4] = 4; +let reducedCalls = 0; +reducedView[1] = { + toString(): string { + if (reducedCalls++ === 0) reduced.length = 1; + return "1"; + }, +}; +reduced.sort(); +console.log(reduced.length, 3 in reduced, 4 in reduced, reduced[1], reduced[2]); +const padded: number[] = [3, 1, 2]; +const paddedView: unknown[] = padded; +paddedView.length = 5; +let paddedCalls = 0; +paddedView[1] = { + toString(): string { + if (paddedCalls++ === 0) padded.push(9); + return "1"; + }, +}; +padded.sort(); +console.log(padded.length, 3 in padded, 4 in padded, padded[5]); +const spread: number[] = [4, 3]; +const spreadView: unknown[] = spread; +let spreadCalls = 0; +spreadView.push({ + toString(): string { + if (spreadCalls++ === 0) { + spreadView[1000] = 1; + for (let i = 0; i < 64; i++) spread.push(i); + } + return "2"; + }, +}); +console.log(spread.toSorted().length, spread.length); +spread.sort(); +console.log(spread.length, spread[0], spread[1], spread[3], 999 in spread, spread[1000]); +const grownMath: number[] = [1]; +const grownMathView: unknown[] = grownMath; +grownMathView[10] = { + valueOf(): number { + grownMathView[5000] = 7; + for (let i = 0; i < 256; i++) grownMath.push(i); + return 3; + }, +}; +console.log(Math.max(...grownMath), grownMath.length); + +// A conversion that removes the element it converts. +const shrinking: number[] = [1]; +const shrinkingView: unknown[] = shrinking; +shrinkingView.push({ + toString: () => { + shrinking.length = 0; + return "gone"; + }, +}); +console.log(shrinking.join("+"), shrinking.length); +const cut: string[] = ["a", "b"]; +const cutView: unknown[] = cut; +cutView.splice(1, 0, { toString: () => ((cut.length = 0), "x") }); +console.log(JSON.stringify(cut.join("-")), cut.length); + +// JSON of elements whose getters change the array or record holding them. +const fillers: number[][] = []; +const rows: number[][] = [[1], [2]]; +const rowView: unknown[] = rows[0]!; +rowView.push({ + get grown() { + for (let i = 0; i < 64; i++) rows.push([i]); + rows.length = 3; + // Reuse the memory the old storage held. + for (let i = 0; i < 64; i++) fillers.push([i, i]); + return true; + }, +}); +console.log(JSON.stringify(rows), rows.length); +const holder = { values: [5] as number[], name: "h" }; +const holderView: unknown[] = holder.values; +holderView.push({ + get replaced() { + holder.values = [9]; + for (let i = 0; i < 64; i++) fillers.push([i]); + return true; + }, +}); +console.log(JSON.stringify(holder), JSON.stringify(holder)); + +// toSorted keeps the full length and reads holes as undefined; indexOf, +// lastIndexOf and includes skip the undefined and foreign elements. +const searched: string[] = ["b", "a"]; +const searchedView: unknown[] = searched; +searchedView[9] = 4; +searchedView[5] = undefined; +console.log(searched.indexOf("b"), searched.lastIndexOf("b"), searched.lastIndexOf("a", 8)); +console.log(searched.includes("a"), searched.lastIndexOf("z")); +const searchedSorted = searched.toSorted(); +console.log(searchedSorted.length, searchedSorted.slice(0, 4), searchedSorted.lastIndexOf("b")); +console.log(searchedSorted.indexOf("a"), searchedSorted.includes("b")); +const counts: number[] = [3, 1]; +const countsView: unknown[] = counts; +countsView[6] = "x"; +countsView[4] = 2; +const countsSorted = counts.toSorted(); +console.log(countsSorted.length, countsSorted, countsSorted.lastIndexOf(3), counts.lastIndexOf(2)); +const far: number[] = [5]; +const farView: unknown[] = far; +farView[70000] = "z"; +const farSorted = far.toSorted(); +console.log(farSorted.length, farSorted.slice(0, 3), farSorted.slice(69999), farSorted.lastIndexOf(5)); diff --git a/tests/corpus/6069-foreign-element-writers.ts b/tests/corpus/6069-foreign-element-writers.ts new file mode 100644 index 000000000..c3accaa09 --- /dev/null +++ b/tests/corpus/6069-foreign-element-writers.ts @@ -0,0 +1,54 @@ +// Values of other types written into typed arrays by code that sees them as +// `any` or `unknown[]`: an `any` parameter, a record field, methods that +// reorder the whole array through the view, and an `unknown[]` parameter. The typed +// array keeps every written value, and the view reads them back. + +function label(values: any): void { + values.push("label"); + values[0] = "first"; +} +const labelled: number[] = [1, 2]; +label(labelled); +console.log(labelled, labelled.length, JSON.stringify(labelled)); + +interface Box { + values: unknown[]; +} +const boxed: number[] = [1, 2, 3]; +const box: Box = { values: boxed }; +box.values[1] = "two"; +box.values.push(null); +console.log(boxed, box.values === boxed); + +// reverse and sort through the view rewrite the typed array. +const filled: number[] = [1, 2, 3, 4]; +const filledView: unknown[] = filled; +filledView[1] = "z"; +filledView[2] = "z"; +console.log(filled); +filledView.reverse(); +console.log(filled, filled[0], filled[3]); +filledView.sort(); +console.log(filled); + +function setAt(values: unknown[], index: number, value: unknown): void { + values[index] = value; +} +const assigned: number[] = [1, 2]; +setAt(assigned, 1, "b"); +setAt(assigned, 3, "d"); +console.log(assigned, assigned.length); + +// The view's own methods and iteration see the same elements. +const shared: number[] = [1, 2]; +const sharedView: unknown[] = shared; +sharedView.push("x"); +console.log(sharedView.map((value) => typeof value).join(",")); +console.log(sharedView.filter((value) => typeof value === "number").length); +for (const value of sharedView) console.log("view", value); +for (const key in shared) console.log("key", key); +console.log(Object.keys(sharedView), sharedView.indexOf("x"), sharedView.includes("x")); + +// Writing a number through the view stores an ordinary element. +sharedView[2] = 3; +console.log(shared, shared[2] + 1); diff --git a/tests/corpus/7515-net-socket-backpressure.ts b/tests/corpus/7515-net-socket-backpressure.ts new file mode 100644 index 000000000..260726a15 --- /dev/null +++ b/tests/corpus/7515-net-socket-backpressure.ts @@ -0,0 +1,136 @@ +// net.Socket write() backpressure, as in Node 24: write answers false when +// the writes not yet completely sent reach the 65536-byte high-water mark, +// each counting in full, which a write made before the connection does; +// writableNeedDrain stays true until 'drain', and 'drain' fires once the +// buffered bytes are sent. The same holds while a custom lookup is pending, +// writableLength counts a large write the kernel took in part in full, +// untyped code reads the same properties, and a write whose send fails +// after the peer reset the connection answers false. An echo client that +// waits for 'drain' when write answers false moves every byte. +import { connect, createServer, Socket } from "node:net"; +import type { LookupAddress, LookupAllOptions, LookupOneOptions } from "node:dns"; + +const CHUNK = 65536; +const TOTAL = 64 * CHUNK; +const chunk = Buffer.alloc(CHUNK, 120); + +function untyped(socket: any): string { + return `${socket.writableLength} ${socket.writableNeedDrain} ${socket.writableHighWaterMark}`; +} + +function echo(port: number): Promise { + return new Promise((resolve) => { + const client = connect(port, "127.0.0.1"); + console.log("small write before connect", client.write("hi"), client.writableLength); + console.log("large write before connect", client.write(Buffer.alloc(70000, 121)), client.writableLength); + console.log("need drain", client.writableNeedDrain, "untyped", untyped(client)); + let sent = 70002; + let received = 0; + const pump = (): void => { + while (sent < TOTAL) { + sent += CHUNK; + if (!client.write(chunk)) { + client.once("drain", pump); + return; + } + } + client.end(); + }; + client.once("drain", () => { + console.log("drain fired, need drain", client.writableNeedDrain, "pending", client.writableLength); + pump(); + }); + client.on("data", (data: Buffer) => { + received += data.length; + }); + client.on("end", () => { + console.log("received", received, "of", sent); + resolve(); + }); + }); +} + +function lookupPending(port: number): Promise { + return new Promise((resolve) => { + const client = connect({ + port, + host: "localhost", + lookup: ( + _host: string, + _options: LookupOneOptions | LookupAllOptions | number | undefined, + callback: (err: NodeJS.ErrnoException | null, address: string | LookupAddress[], family?: number) => void, + ) => { + const answer: LookupAddress[] = [{ address: "127.0.0.1", family: 4 }]; + setTimeout(() => callback(null, answer), 20); + }, + }); + console.log("lookup write", client.write(Buffer.alloc(70000, 97))); + console.log("lookup state", client.writableLength, client.writableNeedDrain, "untyped", untyped(client)); + client.once("drain", () => { + console.log("lookup drain", client.writableLength, client.writableNeedDrain); + client.destroy(); + resolve(); + }); + }); +} + +function largeWrite(port: number): Promise { + return new Promise((resolve) => { + const client = connect(port, "127.0.0.1", () => { + const size = 16 * 1024 * 1024; + console.log("large write", client.write(Buffer.alloc(size, 98))); + console.log("large state", client.writableLength === size, client.writableNeedDrain); + client.once("drain", () => { + console.log("large drain", client.writableLength, client.writableNeedDrain); + client.destroy(); + resolve(); + }); + setTimeout(() => { + if (paused !== null) paused.resume(); + }, 30); + }); + }); +} + +function afterReset(port: number): Promise { + return new Promise((resolve) => { + const client = connect(port, "127.0.0.1", () => { + client.pause(); + setTimeout(() => { + console.log("first write after the peer closed", client.write("a")); + setTimeout(() => { + console.log("write after reset", client.write("b")); + }, 50); + }, 50); + }); + client.on("error", () => {}); + client.on("close", () => { + console.log("closed, destroyed", client.destroyed); + resolve(); + }); + }); +} + +let paused: Socket | null = null; +let mode = "echo"; +const server = createServer((socket: Socket) => { + if (mode === "echo") socket.pipe(socket); + else if (mode === "drain") socket.resume(); + else if (mode === "pause") { + socket.pause(); + paused = socket; + } else socket.destroy(); +}); + +server.listen(0, "127.0.0.1", async () => { + const addr = server.address(); + const port = typeof addr === "object" && addr !== null ? addr.port : 0; + await echo(port); + mode = "drain"; + await lookupPending(port); + mode = "pause"; + await largeWrite(port); + mode = "end"; + await afterReset(port); + server.close(); +}); diff --git a/tests/corpus/7519-net-socket-drain-listener-close.ts b/tests/corpus/7519-net-socket-drain-listener-close.ts new file mode 100644 index 000000000..e1b51e7b4 --- /dev/null +++ b/tests/corpus/7519-net-socket-drain-listener-close.ts @@ -0,0 +1,62 @@ +// 'drain' listeners that capture their socket do not keep it after it +// closes: a once('drain') listener still pending when the peer disconnects +// before the buffer drains, and an on('drain') listener on a socket that +// drained and then closed. The sanitized lane's exit audit checks that +// both sockets are freed. +import { connect, createServer, Socket } from "node:net"; + +function disconnectBeforeDrain(): Promise { + return new Promise((resolve) => { + const server = createServer((socket: Socket) => { + socket.on("error", () => {}); + socket.once("drain", () => console.log("unexpected drain", socket.writableLength)); + console.log("server write", socket.write(Buffer.alloc(16 * 1024 * 1024, 97))); + socket.on("close", () => { + console.log("server socket closed, need drain", socket.writableNeedDrain); + server.close(() => resolve()); + }); + }); + server.listen(0, "127.0.0.1", () => { + const addr = server.address(); + const port = typeof addr === "object" && addr !== null ? addr.port : 0; + const client = connect(port, "127.0.0.1", () => { + client.pause(); + setTimeout(() => client.destroy(), 30); + }); + }); + }); +} + +function drainThenClose(): Promise { + return new Promise((resolve) => { + const server = createServer((socket: Socket) => { + socket.pause(); + setTimeout(() => socket.resume(), 30); + }); + server.listen(0, "127.0.0.1", () => { + const addr = server.address(); + const port = typeof addr === "object" && addr !== null ? addr.port : 0; + const client = connect(port, "127.0.0.1", () => { + let drains = 0; + client.on("drain", () => { + drains++; + console.log("drain", drains, client.writableLength); + client.end(); + }); + console.log("client write", client.write(Buffer.alloc(16 * 1024 * 1024, 98))); + }); + client.on("close", () => { + console.log("client closed"); + server.close(() => resolve()); + }); + }); + }); +} + +async function main(): Promise { + await disconnectBeforeDrain(); + await drainThenClose(); + console.log("done"); +} + +main(); diff --git a/tests/corpus/8010-http-header-raw-bytes.ts b/tests/corpus/8010-http-header-raw-bytes.ts new file mode 100644 index 000000000..d8408f335 --- /dev/null +++ b/tests/corpus/8010-http-header-raw-bytes.ts @@ -0,0 +1,72 @@ +// Header values hold the raw bytes a client sent, one character per byte +// (Node decodes them as latin1). Bytes that are not well-formed UTF-8 (lone +// continuation bytes, runs of them, truncated leads, overlong leads and +// bytes no sequence uses) at every offset within an eight-byte word encode +// to the same bytes as in Node with latin1, ascii, binary, hex, base64 and +// utf16le. Strings built in the program take the same encodings with two-, +// three- and four-byte characters at every offset. +import { createServer } from "node:http"; +import { connect } from "node:net"; + +const patterns: number[][] = [ + [0x80], + [0x80, 0x81, 0xbf, 0x90], + [0xc3], + [0xc0, 0x80, 0xc1, 0xbf], + [0xe9, 0x41, 0xe9], + [0xf0, 0x9f, 0x41], + [0xe2, 0x82], + [0xff, 0xfe, 0xf5], + [0x78, 0xe9, 0x79, 0x80, 0x80, 0x7a, 0xdf], +]; +const encodings = ["latin1", "ascii", "binary", "hex", "base64", "utf16le"] as const; + +function hex(bytes: Buffer): string { + return bytes.toString("hex"); +} + +function describe(label: string, value: string): void { + console.log(label, encodings.map((encoding) => `${encoding}=${hex(Buffer.from(value, encoding))}`).join(" ")); +} + +const names: string[] = []; +const head: Buffer[] = [Buffer.from("GET / HTTP/1.1\r\nHost: x\r\nConnection: close\r\n")]; +for (let p = 0; p < patterns.length; p++) { + for (let offset = 0; offset < 10; offset++) { + for (const tail of ["", "bcdefghij"]) { + const name = `x-${p}-${offset}-${tail.length}`; + names.push(name); + head.push(Buffer.from(`${name}: ${"a".repeat(offset)}`)); + head.push(Buffer.from(patterns[p]!)); + head.push(Buffer.from(`${tail}\r\n`)); + } + } +} +head.push(Buffer.from("\r\n")); + +const server = createServer((req, res) => { + for (const name of names) { + const value = req.headers[name]; + if (typeof value === "string") describe(name, value); + else console.log(name, "missing"); + } + res.end("ok"); +}); + +server.listen(0, "127.0.0.1", () => { + const address = server.address(); + const port = typeof address === "object" && address !== null ? address.port : 0; + const socket = connect({ port, host: "127.0.0.1" }, () => { + socket.write(Buffer.concat(head)); + }); + socket.on("data", () => {}); + socket.on("close", () => { + server.close(); + for (const text of ["é", "éü", "߀", "😀", "é😀ü", "߿\u0080"]) { + for (let offset = 0; offset < 17; offset++) { + const value = "a".repeat(offset) + text + "b".repeat(offset % 9) + text; + describe(`text ${offset} ${JSON.stringify(text)}`, value); + } + } + }); +}); diff --git a/tests/corpus/8011-foreign-element-tojson.ts b/tests/corpus/8011-foreign-element-tojson.ts new file mode 100644 index 000000000..60db9c9bd --- /dev/null +++ b/tests/corpus/8011-foreign-element-tojson.ts @@ -0,0 +1,72 @@ +// JSON.stringify of a typed array runs the toJSON of a foreign element +// (an object written through an unknown[] view) with the element's index +// as its key, as Node does, including when the hook grows, truncates or +// replaces the arrays holding it. A hook or getter that throws stops the +// stringify with its exception. +const nums: number[] = [1, 2, 3]; +const view: unknown[] = nums; +view[1] = { + toJSON(key: string) { + return `v${key}`; + }, +}; +view[2] = { toJSON: 5 }; +console.log(JSON.stringify(nums), JSON.stringify(view)); + +const words: string[] = ["a", "b"]; +const wordsView: unknown[] = words; +wordsView[0] = { toJSON: () => undefined }; +wordsView[1] = { toJSON: () => ({ nested: [1, 2] }) }; +console.log(JSON.stringify(words), JSON.stringify({ words })); + +// The hook grows the parent past its storage and then truncates it. +const outer: number[][] = [[1, 2], [3], [4, 5, 6]]; +const innerView: unknown[] = outer[1]!; +let calls = 0; +innerView[1] = { + toJSON() { + calls++; + for (let i = 0; i < 300; i++) outer.push([i, i + 1]); + if (calls === 1) outer.length = 2; + return "hooked"; + }, +}; +console.log(JSON.stringify(outer), outer.length, calls); + +// The hook replaces the parent's elements two levels up. +const deep: number[][][] = [[[1], [2]], [[3]]]; +const deepView: unknown[] = deep[0]![1]!; +deepView[0] = { + toJSON() { + for (let i = 0; i < 200; i++) deep.push([[i]]); + deep[0] = []; + return 9; + }, +}; +console.log(JSON.stringify(deep).length, deep.length); + +// A throwing hook or getter stops the stringify. +const thrown: number[] = [1, 2]; +const thrownView: unknown[] = thrown; +thrownView[1] = { + toJSON() { + throw new Error("from toJSON"); + }, +}; +try { + console.log(JSON.stringify(thrown)); +} catch (error) { + console.log("caught", (error as Error).message); +} +thrownView[1] = { + get a() { + throw new Error("from getter"); + }, +}; +try { + console.log(JSON.stringify(thrown)); +} catch (error) { + console.log("caught", (error as Error).message); +} +const after: unknown[] = [1, { b: 2 }]; +console.log(JSON.stringify(after)); diff --git a/tests/corpus/8165-rsa-keygen-lengths.cjs b/tests/corpus/8165-rsa-keygen-lengths.cjs new file mode 100644 index 000000000..35e2cb28f --- /dev/null +++ b/tests/corpus/8165-rsa-keygen-lengths.cjs @@ -0,0 +1,40 @@ +// RSA key generation within OpenSSL's limits: any modulus length from +// 512 bits (an odd length from 2048 bits yields one bit less, as +// OpenSSL's SP 800-56B generation does) and odd exponents up to 2^32 - 1. +"use strict"; +const crypto = require("crypto"); + +const cases = [ + [512, 65537], + [513, 65537], + [1023, 3], + [1024, 0x10001], + [2049, 65537], + [768, 4294967295], + [520, 17], +]; +for (const [modulusLength, publicExponent] of cases) { + const { publicKey, privateKey } = crypto.generateKeyPairSync("rsa", { modulusLength, publicExponent }); + const details = privateKey.asymmetricKeyDetails; + const data = Buffer.from("rsa key lengths"); + const signature = crypto.sign("sha256", data, privateKey); + console.log( + modulusLength, + details.modulusLength, + details.publicExponent, + publicKey.asymmetricKeyDetails.modulusLength, + signature.length, + crypto.verify("sha256", data, publicKey, signature), + ); +} +for (const [modulusLength, publicExponent] of [[511, 65537], [1024, 1], [1024, 4], [1024, 0]]) { + try { + crypto.generateKeyPairSync("rsa", { modulusLength, publicExponent }); + console.log(modulusLength, publicExponent, "generated"); + } catch (error) { + console.log(modulusLength, publicExponent, error.message); + } +} +crypto.generateKeyPair("rsa", { modulusLength: 515, publicExponent: 3 }, (error, publicKey) => { + console.log("async", error, publicKey.asymmetricKeyDetails.modulusLength); +}); diff --git a/tests/corpus/8166-rsa-pss-keys.cjs b/tests/corpus/8166-rsa-pss-keys.cjs new file mode 100644 index 000000000..aad90f6aa --- /dev/null +++ b/tests/corpus/8166-rsa-pss-keys.cjs @@ -0,0 +1,104 @@ +// RSA-PSS keys (id-RSASSA-PSS): generation with and without +// RSASSA-PSS-params, their details, SubjectPublicKeyInfo and PKCS#8 in +// OpenSSL's encoding, import of both, and the restrictions on signatures: +// PSS padding only, the key's digest and MGF1 digest, and at least its +// salt length. The deprecated hash and mgf1Hash options run in a child +// process, whose DEP0154 reports are printed with process ids normalized. +"use strict"; +const { spawnSync } = require("child_process"); +const c = require("crypto"); +const fmt = (v) => JSON.stringify(v, (k, x) => (typeof x === "bigint" ? `${x}n` : x)); +const tryIt = (label, fn) => { + try { + const r = fn(); + console.log(label, typeof r === "string" ? r.split("\n")[0] : Buffer.isBuffer(r) ? `buffer ${r.length > 0}` : fmt(r)); + } catch (e) { + console.log(label, e.name, e.code, e.message); + } +}; +const algOf = (spki) => { + // The AlgorithmIdentifier: the SEQUENCE after the outer header. + const start = spki[1] & 0x80 ? 2 + (spki[1] & 0x7f) : 2; + const len = spki[start + 1]; + return spki.subarray(start, start + 2 + len).toString("hex"); +}; +const report = (label, options) => tryIt(label, () => c.generateKeyPairSync("rsa-pss", { modulusLength: 512, ...options }).publicKey.asymmetricKeyDetails); + +function main() { + const cases = { + none: {}, + sha256: { hashAlgorithm: "sha256" }, + full: { hashAlgorithm: "sha256", mgf1HashAlgorithm: "sha384", saltLength: 16 }, + sha1: { hashAlgorithm: "sha1" }, + salt: { saltLength: 10 }, + s512: { hashAlgorithm: "sha512", saltLength: 0 }, + mgf: { mgf1HashAlgorithm: "sha256" }, + }; + for (const [name, options] of Object.entries(cases)) { + const { publicKey, privateKey } = c.generateKeyPairSync("rsa-pss", { modulusLength: ["full", "sha1", "salt", "mgf"].includes(name) ? 512 : 1024, ...options }); + const spki = publicKey.export({ format: "der", type: "spki" }); + const pkcs8 = privateKey.export({ format: "der", type: "pkcs8" }); + console.log(name, publicKey.asymmetricKeyType, fmt(publicKey.asymmetricKeyDetails), fmt(privateKey.asymmetricKeyDetails)); + console.log(name, "spki alg", algOf(spki), "pkcs8 alg", pkcs8.indexOf(Buffer.from(algOf(spki), "hex")) > 0); + tryIt(`${name} pkcs1`, () => privateKey.export({ format: "pem", type: "pkcs1" })); + tryIt(`${name} jwk`, () => publicKey.export({ format: "jwk" })); + tryIt(`${name} pem`, () => privateKey.export({ format: "pem", type: "pkcs8" })); + const reimported = c.createPublicKey(publicKey.export({ format: "pem", type: "spki" })); + const fromPrivate = c.createPublicKey(privateKey.export({ format: "pem", type: "pkcs8" })); + const privateAgain = c.createPrivateKey({ key: pkcs8, format: "der", type: "pkcs8" }); + console.log(name, reimported.equals(publicKey), fromPrivate.equals(publicKey), privateAgain.asymmetricKeyType); + const data = Buffer.from("rsa-pss"); + const signature = c.sign(null, data, privateAgain); + console.log(name, c.verify(null, data, publicKey, signature), c.verify(null, data, reimported, signature)); + } + + const data = Buffer.from("restricted"); + const { publicKey, privateKey } = c.generateKeyPairSync("rsa-pss", { + modulusLength: 512, + hashAlgorithm: "sha256", + saltLength: 16, + }); + tryIt("sign sha1", () => c.sign("sha1", data, privateKey)); + tryIt("sign pkcs1", () => c.sign("sha256", data, { key: privateKey, padding: c.constants.RSA_PKCS1_PADDING })); + tryIt("sign salt 8", () => c.sign("sha256", data, { key: privateKey, saltLength: 8 })); + tryIt("verify salt 8", () => c.verify("sha256", data, { key: publicKey, saltLength: 8 }, Buffer.alloc(64))); + for (const saltLength of [16, 20, 30]) { + const signature = c.sign("sha256", data, { key: privateKey, saltLength }); + console.log( + saltLength, + c.verify("sha256", data, publicKey, signature), + c.verify("sha256", data, { key: publicKey, saltLength: c.constants.RSA_PSS_SALTLEN_AUTO }, signature), + ); + } + const signer = c.createSign("sha256").update(data); + console.log("createVerify", c.createVerify("sha256").update(data).verify(publicKey, signer.sign(privateKey))); + const free = c.generateKeyPairSync("rsa-pss", { modulusLength: 1024 }); + const freeSignature = c.sign("sha512", data, free.privateKey); + console.log("free sha512", c.verify("sha512", data, free.publicKey, freeSignature)); + + report("bad digest", { hashAlgorithm: "nope" }); + report("bad mgf1", { mgf1HashAlgorithm: "nope" }); + report("negative salt", { saltLength: -1 }); + report("upper", { hashAlgorithm: "SHA256" }); + report("alias", { hashAlgorithm: "RSA-SHA256" }); +} + +if (process.argv[2] === "deprecated") { + report("hash", { hash: "sha256" }); + report("conflict", { hashAlgorithm: "sha256", hash: "sha1" }); + report("mgf1Hash", { hashAlgorithm: "sha256", mgf1Hash: "sha384" }); + report("hash type", { hash: 5 }); + // Node drops deprecations while noDeprecation is set when it reports + // them, so the flag is set once the reports above are out. + setTimeout(() => { + process.noDeprecation = true; + report("quiet", { hash: "sha512" }); + console.log("end of turn"); + }); +} else { + main(); + const child = spawnSync(process.execPath, [process.argv[1], "deprecated"], { encoding: "utf8" }); + console.log(`status: ${child.status}`); + console.log(child.stdout); + console.log(child.stderr.replace(/\(node:\d+\)/g, "(node:)")); +} diff --git a/tests/corpus/8167-encrypted-private-keys.cjs b/tests/corpus/8167-encrypted-private-keys.cjs new file mode 100644 index 000000000..7b274e500 --- /dev/null +++ b/tests/corpus/8167-encrypted-private-keys.cjs @@ -0,0 +1,143 @@ +// Encrypted private keys: PKCS#8 as PBES2 and PKCS#1 and SEC1 as +// traditional PEM with the CBC and ECB ciphers, import with a passphrase, +// generateKeyPair with a cipher, and OpenSSL's errors for missing and +// wrong passphrases, unknown ciphers, and ciphers a form refuses. +"use strict"; +const c = require("crypto"); +const show = (e) => + `${e.name} ${e.code} ${e.message}${e.library ? ` [${e.library}: ${e.reason}]` : ""}${ + e.opensslErrorStack ? ` ${JSON.stringify(e.opensslErrorStack)}` : "" + }`; +const tryIt = (label, fn) => { + try { + console.log(label, fn()); + } catch (e) { + console.log(label, show(e)); + } +}; +// The PEM label and headers, without the random IV. +const head = (pem) => + pem + .split("\n") + .filter((l) => l.startsWith("-----BEGIN") || l.includes(":")) + .map((l) => l.replace(/,[0-9A-F]+$/, "")) + .join(" | "); + +const ec = c.generateKeyPairSync("ec", { namedCurve: "P-256" }); +const rsa = c.generateKeyPairSync("rsa", { modulusLength: 1024 }); +const same = (key, pair) => c.createPublicKey(key).equals(pair.publicKey); + +for (const cipher of ["aes-128-cbc", "aes-192-cbc", "aes-256-cbc", "aes-128-ecb", "aes-256-ecb", "des-ede3-cbc", "aes256", "AES-128-CBC", "2.16.840.1.101.3.4.1.42"]) { + for (const format of ["pem", "der"]) { + tryIt(`pkcs8 ${format} ${cipher}`, () => { + const out = ec.privateKey.export({ type: "pkcs8", format, cipher, passphrase: "top secret" }); + const key = c.createPrivateKey({ key: out, format, type: format === "der" ? "pkcs8" : undefined, passphrase: "top secret" }); + return `${format === "pem" ? head(out) : Buffer.isBuffer(out)} ${key.asymmetricKeyType} ${same(key, ec)}`; + }); + } +} +for (const cipher of ["aes-128-cbc", "aes-256-cbc", "des-ede3-cbc", "des3"]) { + tryIt(`sec1 ${cipher}`, () => { + const out = ec.privateKey.export({ type: "sec1", format: "pem", cipher, passphrase: Buffer.from("pw") }); + return `${head(out)} ${same(c.createPrivateKey({ key: out, passphrase: "pw" }), ec)}`; + }); + tryIt(`pkcs1 ${cipher}`, () => { + const out = rsa.privateKey.export({ type: "pkcs1", format: "pem", cipher, passphrase: "pw" }); + return `${head(out)} ${same(c.createPrivateKey({ key: out, passphrase: Buffer.from("pw") }), rsa)}`; + }); +} +tryIt("empty passphrase", () => { + const out = rsa.privateKey.export({ type: "pkcs8", format: "pem", cipher: "aes-128-cbc", passphrase: "" }); + return same(c.createPrivateKey({ key: out, passphrase: "" }), rsa); +}); + +// Reading encrypted keys. These are fixed encryptions of one P-256 key +// with the passphrase "pw": decrypting with a wrong passphrase gives +// random-looking bytes, which a fresh encryption would make fail at +// different steps from run to run. +const fixedKey = [ + "-----BEGIN PRIVATE KEY-----", + "MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgExcc5/xZ3vYL2Dpg", + "XnB7E8gQhcW1bkUcIGJWBGvrrGChRANCAAR0wxNUH1yo/U9nqFY4Gt5SVzPjvWq6", + "NzlStUC6Njk9BdR7SvBlNIGDAFlOfMwhfZbvvvMP+rLMR3RtBg3Ketxe", + "-----END PRIVATE KEY-----", +].join("\n") + "\n"; +const p8pem = [ + "-----BEGIN ENCRYPTED PRIVATE KEY-----", + "MIH0MF8GCSqGSIb3DQEFDTBSMDEGCSqGSIb3DQEFDDAkBBBkpqs2611cRjGDZTd4", + "FmNeAgIIADAMBggqhkiG9w0CCQUAMB0GCWCGSAFlAwQBKgQQRhL9yxS4i0ngtke+", + "frhURASBkN0dk+GQNCGU5cFC+MMswOr9XwGaQnLbJo7VnhCje3//JaVJDjexrd69", + "YZliX7pKGNghfFf+eBAeOAkwbAK39T8KQPwTJVJ+Z5O485N2fl5dPlr3gaEA3+cm", + "LQgasmL7Zv9UOH2iwvWllFYL3bnwr7VsqTCW2xe53qTB+dWVurP0cCiZNGEnUzrf", + "4bvWNrW0nA==", + "-----END ENCRYPTED PRIVATE KEY-----", +].join("\n") + "\n"; +const p8der = Buffer.from("MIH0MF8GCSqGSIb3DQEFDTBSMDEGCSqGSIb3DQEFDDAkBBAOrVavYYEvy3KUHwK0hbQnAgIIADAMBggqhkiG9w0CCQUAMB0GCWCGSAFlAwQBKgQQmEnqHOLegUirb49LD8oNhASBkGUwe3SZciScH/tWuoK5JWujckNHRKROwu81a5AGjIKxkvqCm+Qp9Gr4rPsb2vbv3bgfojbEKvSGvtuqB/X/jpN5UMZwWI3D9Ocd1+jEq1FqDb3rb3qxcoO9jmwuFFzNeYy4sW29XFuKTxu4aJHT1sEtMzIVlRLv53aXzpp9eQ9j47EOTWFqIpkt02TRFuazCQ==", "base64"); +const sec1 = [ + "-----BEGIN EC PRIVATE KEY-----", + "Proc-Type: 4,ENCRYPTED", + "DEK-Info: AES-128-CBC,E73A95A6783C45F914B397FE018521D2", + "", + "nsuhQFMPr+QFITcpmr2JWs+Zep4wxoCWMEk1jzyyXSJddfebIXOFg4xmtK8YJXIy", + "vqa6pvAIiak0iEZjVSn20+JAg2wqQ0sLaFPhBmhcZk+xuQoLD8CcYvROO5VtVELm", + "DkgOUcJ1oEfZJIRM6U2oDNFmj2NB/SnSQUf1niy7ba8=", + "-----END EC PRIVATE KEY-----", +].join("\n") + "\n"; +const fixedPublic = c.createPublicKey(fixedKey); +tryIt("public from encrypted", () => c.createPublicKey({ key: p8pem, passphrase: "pw" }).equals(fixedPublic)); +tryIt("public from traditional", () => c.createPublicKey({ key: sec1, passphrase: "pw" }).type); +tryIt("pkcs8 pem no passphrase", () => c.createPrivateKey(p8pem)); +tryIt("pkcs8 der no passphrase", () => c.createPrivateKey({ key: p8der, format: "der", type: "pkcs8" })); +tryIt("sec1 no passphrase", () => c.createPrivateKey(sec1)); +tryIt("public no passphrase", () => c.createPublicKey(p8pem)); +tryIt("pkcs8 pem wrong", () => c.createPrivateKey({ key: p8pem, passphrase: "nope" })); +tryIt("pkcs8 der wrong", () => c.createPrivateKey({ key: p8der, format: "der", type: "pkcs8", passphrase: "nope" })); +tryIt("sec1 wrong", () => c.createPrivateKey({ key: sec1, passphrase: "nope" })); +tryIt("passphrase type", () => c.createPrivateKey({ key: p8pem, passphrase: 5 })); +tryIt("dek unknown", () => c.createPrivateKey({ key: sec1.replace(/DEK-Info: AES-128-CBC/, "DEK-Info: NOPE-CBC"), passphrase: "pw" })); +tryIt("dek legacy", () => + c.createPrivateKey({ key: sec1.replace(/DEK-Info: AES-128-CBC,([0-9A-F]{16})[0-9A-F]+/, "DEK-Info: DES-CBC,$1"), passphrase: "pw" }), +); +tryIt("dek legacy no passphrase", () => + c.createPrivateKey(sec1.replace(/DEK-Info: AES-128-CBC,([0-9A-F]{16})[0-9A-F]+/, "DEK-Info: DES-CBC,$1")), +); + +// Export options. +const opts = (extra) => ({ type: "pkcs8", format: "pem", ...extra }); +tryIt("unknown cipher", () => ec.privateKey.export(opts({ cipher: "nope", passphrase: "x" }))); +tryIt("cipher without passphrase", () => ec.privateKey.export(opts({ cipher: "aes-128-cbc" }))); +tryIt("passphrase without cipher", () => ec.privateKey.export(opts({ passphrase: "x" }))); +tryIt("cipher type", () => ec.privateKey.export(opts({ cipher: 5, passphrase: "x" }))); +tryIt("sec1 der", () => ec.privateKey.export({ type: "sec1", format: "der", cipher: "aes-128-cbc", passphrase: "x" })); +tryIt("pkcs1 der", () => rsa.privateKey.export({ type: "pkcs1", format: "der", cipher: "aes-128-cbc", passphrase: "x" })); +// OpenSSL refuses these ciphers for PKCS#8, and the last ones for +// traditional PEM too (the P-256 key's PKCS#8 and SEC1 lengths are not +// multiples of 8, which key wrap needs). +for (const cipher of ["aes-128-gcm", "aes-256-ccm", "aes-128-ctr", "chacha20-poly1305", "aes-128-xts", "des-ede3-cfb", "2.16.840.1.101.3.4.1.6"]) + tryIt(`refused pkcs8 ${cipher}`, () => ec.privateKey.export(opts({ cipher, passphrase: "x" })).length > 0); +for (const cipher of ["aes-128-wrap", "id-aes256-wrap", "des-cbc", "rc4"]) { + tryIt(`refused pkcs8 ${cipher}`, () => ec.privateKey.export(opts({ cipher, passphrase: "x" })).length > 0); + tryIt(`refused sec1 ${cipher}`, () => ec.privateKey.export({ type: "sec1", format: "pem", cipher, passphrase: "x" })); +} +for (const cipher of ["aes-128-ecb", "aes-256-ecb", "des-ede3"]) + tryIt(`refused sec1 ${cipher}`, () => ec.privateKey.export({ type: "sec1", format: "pem", cipher, passphrase: "x" })); + +// generateKeyPair with a cipher. +const encoding = (cipher, type = "pkcs8") => ({ + namedCurve: "P-256", + publicKeyEncoding: { type: "spki", format: "pem" }, + privateKeyEncoding: { type, format: "pem", cipher, passphrase: "gen" }, +}); +tryIt("keygen sync", () => { + const { publicKey, privateKey } = c.generateKeyPairSync("ec", encoding("aes-256-cbc")); + return `${head(privateKey)} ${c.createPublicKey({ key: privateKey, passphrase: "gen" }).equals(c.createPublicKey(publicKey))}`; +}); +tryIt("keygen unknown", () => c.generateKeyPairSync("ec", encoding("nope"))); +tryIt("keygen gcm", () => c.generateKeyPairSync("ec", encoding("aes-128-gcm"))); +tryIt("keygen sec1 ecb", () => c.generateKeyPairSync("ec", encoding("aes-128-ecb", "sec1"))); +// One job at a time: Node's jobs finish in either order. +c.generateKeyPair("ec", encoding("des-ede3-cbc", "sec1"), (err, publicKey, privateKey) => { + console.log("keygen async", err, head(privateKey)); + console.log("keygen async key", c.createPublicKey({ key: privateKey, passphrase: "gen" }).equals(c.createPublicKey(publicKey))); + c.generateKeyPair("ec", encoding("aes-128-ccm"), (err) => console.log("keygen async ccm", show(err))); +}); diff --git a/tests/corpus/8178-rsa-pss-auto-min-salt.cjs b/tests/corpus/8178-rsa-pss-auto-min-salt.cjs new file mode 100644 index 000000000..78e288591 --- /dev/null +++ b/tests/corpus/8178-rsa-pss-auto-min-salt.cjs @@ -0,0 +1,59 @@ +// Verifying with RSA_PSS_SALTLEN_AUTO against an RSA-PSS key whose +// parameters set a minimum salt length recovers the salt length from the +// signature and does not enforce that minimum, as in Node: a valid +// signature with a shorter salt verifies. The default salt length (the +// key's minimum) accepts only that length. The keys use +// an MGF1 digest equal to the message digest and a different one. The +// stricter public key is the generated one with its minimum salt length +// raised from 16 to 32 in its DER. +const c = require("node:crypto"); + +const data = Buffer.from("restricted salt"); + +function raiseMinimumSalt(publicKey) { + const der = publicKey.export({ format: "der", type: "spki" }); + const from = Buffer.from([0xa2, 0x03, 0x02, 0x01, 0x10]); + const at = der.indexOf(from); + if (at < 0) throw new Error("no saltLength field"); + der[at + 4] = 0x20; + return c.createPublicKey({ key: der, format: "der", type: "spki" }); +} + +function attempt(label, fn) { + try { + console.log(label, fn()); + } catch (e) { + console.log(label, "throws", e.code, e.message); + } +} + +for (const mgf1HashAlgorithm of ["sha256", "sha384"]) { + const { privateKey, publicKey } = c.generateKeyPairSync("rsa-pss", { + modulusLength: 1024, + hashAlgorithm: "sha256", + mgf1HashAlgorithm, + saltLength: 16, + }); + const strict = raiseMinimumSalt(publicKey); + const { mgf1HashAlgorithm: mgf1, saltLength: minimum } = strict.asymmetricKeyDetails; + console.log(mgf1HashAlgorithm, "strict key", mgf1, minimum); + for (const saltLength of [16, 20, 32, 40]) { + const signature = c.sign("sha256", data, { key: privateKey, saltLength }); + const label = `${mgf1HashAlgorithm} salt ${saltLength}`; + attempt(`${label} auto, key minimum 16:`, () => + c.verify("sha256", data, { key: publicKey, saltLength: c.constants.RSA_PSS_SALTLEN_AUTO }, signature), + ); + attempt(`${label} auto, key minimum 32:`, () => + c.verify("sha256", data, { key: strict, saltLength: c.constants.RSA_PSS_SALTLEN_AUTO }, signature), + ); + attempt(`${label} default, key minimum 32:`, () => c.verify("sha256", data, strict, signature)); + attempt(`${label} digest length, key minimum 32:`, () => + c.verify("sha256", data, { key: strict, saltLength: c.constants.RSA_PSS_SALTLEN_DIGEST }, signature), + ); + const v = c.createVerify("sha256"); + v.update(data); + attempt(`${label} createVerify auto, key minimum 32:`, () => + v.verify({ key: strict, saltLength: c.constants.RSA_PSS_SALTLEN_AUTO }, signature), + ); + } +} diff --git a/tests/corpus/8183-key-passphrase-encoding.cjs b/tests/corpus/8183-key-passphrase-encoding.cjs new file mode 100644 index 000000000..e52fa6c1c --- /dev/null +++ b/tests/corpus/8183-key-passphrase-encoding.cjs @@ -0,0 +1,28 @@ +// A string passphrase is decoded with the key's encoding, as in Node: +// createPrivateKey's { encoding } and the encoding option of an encrypted +// export both apply to it. +const c = require("node:crypto"); + +const { privateKey } = c.generateKeyPairSync("ec", { namedCurve: "P-256" }); +const pass = "pässwörd"; +const hex = Buffer.from(pass).toString("hex"); +const pem = privateKey.export({ format: "pem", type: "pkcs8", cipher: "aes-256-cbc", passphrase: hex, encoding: "hex" }); +const der = privateKey.export({ format: "der", type: "pkcs8", cipher: "aes-128-cbc", passphrase: pass }); + +function attempt(label, options) { + try { + const key = c.createPrivateKey(options); + console.log(label, key.asymmetricKeyType, key.equals(privateKey)); + } catch (e) { + console.log(label, "throws", e.code); + } +} + +attempt("utf8 passphrase", { key: pem, passphrase: pass }); +attempt("hex passphrase, encoding hex", { key: Buffer.from(pem), passphrase: hex, encoding: "hex" }); +attempt("base64 passphrase, encoding base64", { key: Buffer.from(pem), passphrase: Buffer.from(pass).toString("base64"), encoding: "base64" }); +attempt("latin1 passphrase, encoding latin1", { key: Buffer.from(pem), passphrase: Buffer.from(pass).toString("latin1"), encoding: "latin1" }); +attempt("hex passphrase, no encoding", { key: pem, passphrase: hex }); +attempt("Buffer passphrase, encoding hex", { key: Buffer.from(pem), passphrase: Buffer.from(pass), encoding: "hex" }); +attempt("der, hex passphrase, encoding hex", { key: der, format: "der", type: "pkcs8", passphrase: hex, encoding: "hex" }); +attempt("hex key string, encoding hex", { key: Buffer.from(pem).toString("hex"), passphrase: hex, encoding: "hex" }); diff --git a/tests/corpus/8240-asserted-callee-errors.ts b/tests/corpus/8240-asserted-callee-errors.ts new file mode 100644 index 000000000..91537ed70 --- /dev/null +++ b/tests/corpus/8240-asserted-callee-errors.ts @@ -0,0 +1,25 @@ +// A non-null assertion is erased in JavaScript, so calling an asserted callee +// that holds undefined throws the call's own TypeError, with V8's spelling of +// the callee: element keys, literal keys as names, parentheses and asserted +// links dropped, call results as f(...). +type F = (x: number) => number; +const table: Record = { inc: (x) => x + 1 }; +const obj: { cb?: F; inner?: { cb?: F } } = { inner: {} }; +const arr: (F | undefined)[] = [undefined]; +let maybe: F | undefined; +function get(): F | undefined { return undefined; } +const key = process.argv.length > 99 ? "inc" : "missing"; +const cases: [string, () => number][] = [ + ["computed", () => table[key]!(1)], + ["literal key", () => table["missing"]!(1)], + ["dotted", () => obj.cb!(1)], + ["nested", () => obj.inner!.cb!(1)], + ["element", () => arr[0]!(1)], + ["local", () => maybe!(1)], + ["parens", () => (table[key])!(1)], + ["call result", () => get()!(1)], + ["parens outside", () => (table[key]!)(1)], +]; +for (const [name, run] of cases) { + try { console.log(name, run()); } catch (e) { console.log(name, (e as Error).name + ": " + (e as Error).message); } +} diff --git a/tests/corpus/8347-string-iteration-code-points.ts b/tests/corpus/8347-string-iteration-code-points.ts new file mode 100644 index 000000000..3a1286140 --- /dev/null +++ b/tests/corpus/8347-string-iteration-code-points.ts @@ -0,0 +1,96 @@ +// The string iterator: for-of, spread, Array.from, yield* and +// [Symbol.iterator]() walk code points. Astral characters come back whole, +// a lone surrogate is a one-unit character, and halves joined by +// concatenation form one character, as in Node. +function unitCodes(c: string): string { + const codes: string[] = []; + for (let k = 0; k < c.length; k++) codes.push(c.charCodeAt(k).toString(16)); + return `${c.length}:${codes.join(".")}`; +} +const show = (chars: string[]): string => chars.map(unitCodes).join(" "); + +const samples = [ + "", + "abc", + "aé😀b", + "\u0000\u007f\u0080߿ࠀ￿", + "\u{10000}\u{10ffff}", + "é ñ", + "\ud800", + "\udc00", + "x\ud83d", + "\ude00x", + "\udc00\ud800", + "\ud83d" + "\ude00", + "a\ud83d" + "\ude00b\ud83d", + "日本語テキスト", +]; +for (const s of samples) { + const looped: string[] = []; + for (const c of s) looped.push(c); + console.log(JSON.stringify(s), show(looped), show([...s]), show(Array.from(s)), [...s].length, s.length); +} + +// Loop control, heads and bindings. +const text = "a😀b\ud800c"; +let out = ""; +outer: for (const c of text) { + if (c === "b") continue; + for (const d of "xy") { + if (d === "y" && c === "c") continue outer; + out += c + d; + } + if (c === "\ud800") out += "|"; +} +console.log(out); +for (var v of "hé") out += v; +console.log(out); +let last = ""; +for (last of "q😀") out += last.length; +console.log(out, last); +for (const [first] of ["😀z", "ab"].join("")) out += first; +console.log(out); +let source = "abc"; +for (const c of source) { + source += c; + out += source.length; +} +console.log(out, source); +let n = 0; +for (const c of "stop😀here") { + if (c === "😀") break; + n += c.length; +} +console.log(n); + +// Characters as values: lengths, Map keys and code points. +const counts = new Map(); +for (const c of "abracadabra😀😀é") counts.set(c, (counts.get(c) ?? 0) + 1); +console.log([...counts].map(([k, v]) => `${k}=${v}`).join(",")); +let points = 0; +for (const c of "😀é\ud800a") points += c.codePointAt(0)!; +console.log(points); + +// Generators and explicit iterators. +function* letters(s: string): Generator { + yield* s; +} +console.log(show([...letters("x😀\udc00y")])); +const it = "o😀"[Symbol.iterator](); +let step = it.next(); +while (!step.done) { + out += step.value.length; + step = it.next(); +} +console.log(out); + +// A long string built from mixed pieces. +let long = ""; +for (let i = 0; i < 2000; i++) long += i % 3 === 0 ? "a" : i % 3 === 1 ? "é" : "😀"; +let units = 0; +let chars = 0; +for (const c of long) { + units += c.length; + chars++; +} +console.log(units, chars, long.length, [...long].length); diff --git a/tests/corpus/8348-string-iteration-worker-characters.ts b/tests/corpus/8348-string-iteration-worker-characters.ts new file mode 100644 index 000000000..52a38fb78 --- /dev/null +++ b/tests/corpus/8348-string-iteration-worker-characters.ts @@ -0,0 +1,44 @@ +import { Worker, isMainThread, parentPort } from "node:worker_threads"; + +// The one-character strings of U+0080..U+07FF that string iteration returns +// are shared by every thread. A worker that is the first to iterate them +// and exits without sharing anything else leaves them in use: the main +// thread iterates the same characters after the worker's memory goes back. + +const text = "aé ñ ß ü Ω ж א ع ߿"; + +function codes(s: string): string { + const out: string[] = []; + for (const c of s) out.push(`${c.length}:${c.charCodeAt(0).toString(16)}`); + return out.join(" "); +} + +function runWorker(): Promise { + return new Promise((resolve, reject) => { + const worker = new Worker(new URL(import.meta.url)); + let count = -1; + worker.on("message", (value: number) => { + count = value; + }); + worker.on("error", reject); + worker.on("exit", () => resolve(count)); + }); +} + +async function main(): Promise { + console.log("worker", await runWorker()); + // The worker's memory goes back shortly after it exits. + await new Promise((resolve) => setTimeout(resolve, 300)); + console.log("main", codes(text)); + console.log("spread", [...text].join("|")); + console.log("from", Array.from("ñandú").map((c) => c.length).join(",")); +} + +if (isMainThread) { + void main(); +} else { + let count = 0; + for (const c of text) count += c.length; + count += [...text].length + Array.from("ñandú").length; + parentPort!.postMessage(count); +} diff --git a/tests/corpus/8351-regex-literal-replacers.ts b/tests/corpus/8351-regex-literal-replacers.ts new file mode 100644 index 000000000..12968d4d9 --- /dev/null +++ b/tests/corpus/8351-regex-literal-replacers.ts @@ -0,0 +1,72 @@ +// Global substitutions of regex literals with literal replacements: matches +// at every offset around 16-character blocks and at the end of the +// subject, empty matches, alternation order, case-insensitive and +// multiline flags, word boundaries, replacements that are empty, longer +// than the match or hold quotes and backslashes, outputs that grow far past +// the subject, subjects without a match, and subjects outside ASCII (which +// take the general loop). replace and replaceAll. +const substitutions: [string, (s: string) => string][] = [ + ["<[^>]*>", (s) => s.replace(/<[^>]*>/g, "|")], + ["a[NSt]|BY", (s) => s.replace(/a[NSt]|BY/g, "<2>")], + ["tHa[Nt]", (s) => s.replace(/tHa[Nt]/g, "<4>")], + ["\\|[^|][^|]*\\|", (s) => s.replace(/\|[^|][^|]*\|/g, "-")], + ["x*", (s) => s.replace(/x*/g, "-")], + ["\\b", (s) => s.replace(/\b/g, "^")], + ["^ab", (s) => s.replace(/^ab/gm, "Z")], + ["ab|abc|b", (s) => s.replace(/ab|abc|b/g, "#")], + ["AB (i)", (s) => s.replace(/AB/gi, "")], + ["(a)(b)?", (s) => s.replace(/(a)(b)?/g, "q")], + ["\\d+", (s) => s.replace(/\d+/g, '"\\\n')], + ["[^a-z]", (s) => s.replace(/[^a-z]/g, "")], + ["a", (s) => s.replace(/a/g, "<<<>>>")], + ["cat (replaceAll)", (s) => s.replaceAll(/cat/g, "dog")], + ["[aeiou]{2} (replaceAll)", (s) => s.replaceAll(/[aeiou]{2}/g, "~")], +]; + +function hash(text: string, h: number): number { + for (let i = 0; i < text.length; i++) h = (h * 31 + text.charCodeAt(i)) >>> 0; + return h; +} + +const filler = "qwertyuiop asdfghjkl zxcvbnm"; +const needles = ["<2>", "", "aN", "BY", "tHat", "|ab|", "||", "xx", "ab", "abc", "AB", "123", "cat", "aeio", "a"]; +let checksum = 0; +let lines = 0; +for (let length = 0; length <= 70; length += 7) { + for (const needle of needles) { + for (const offset of [0, 1, 14, 15, 16, 17, 31, 32, 33]) { + if (offset + needle.length > length) continue; + let subject = ""; + while (subject.length < length) subject += filler; + subject = subject.slice(0, offset) + needle + subject.slice(offset + needle.length, length); + for (const [name, sub] of substitutions) { + const out = sub(subject); + checksum = hash(out, checksum); + lines++; + if (length === 35 && (offset === 15 || offset === 16)) console.log(name, offset, JSON.stringify(out)); + } + } + } +} +console.log("lines", lines, "checksum", checksum); + +// Dense matches, matches that end the subject, and an output many times +// larger than the subject. +const dense = "<1><22><333>".repeat(40) + "tail"; +for (const [name, sub] of substitutions) console.log(name, JSON.stringify(sub(dense).slice(-40)), sub(dense).length); +const many = "a".repeat(5000); +console.log("grow", many.replace(/a/g, "<<<>>>").length, many.replace(/a/g, "<<<>>>").slice(4990)); +console.log("empty", JSON.stringify("".replace(/x*/g, "-")), JSON.stringify("".replace(/a/g, "b"))); +console.log("none", "zzzz".replace(/a[bc]/g, "!"), "zzzz".replaceAll(/a[bc]/g, "!")); + +// Subjects outside ASCII run the general loop with the same results. +for (const subject of ["héllo wörld ", "日本<語>", "a\u{1F600}b", "cat café cat"]) { + console.log(JSON.stringify(subject.replace(/<[^>]*>/g, "|")), JSON.stringify(subject.replaceAll(/cat/g, "dog")), JSON.stringify(subject.replace(/x*/g, "-"))); +} + +// A long DNA-like subject, as in regex-redux. +let dna = ""; +for (let i = 0; i < 4000; i++) dna += "acgt"[(i * 7 + (i >> 2)) % 4] + (i % 61 === 0 ? "tHaN" : i % 37 === 0 ? "aNBY" : ""); +const step1 = dna.replace(/tHa[Nt]/g, "<4>").replace(/a[NSt]|BY/g, "<2>"); +const step2 = step1.replace(/<[^>]*>/g, "|").replace(/\|[^|][^|]*\|/g, "-"); +console.log("dna", dna.length, step1.length, step2.length, hash(step2, 0)); diff --git a/tests/corpus/intersection-restated-members.ts b/tests/corpus/intersection-restated-members.ts new file mode 100644 index 000000000..249b9ddfb --- /dev/null +++ b/tests/corpus/intersection-restated-members.ts @@ -0,0 +1,82 @@ +// An intersection that restates a member with a separately written but +// structurally equal type maps as that one type. Restating an optional member +// as required makes it required; the checker keeps both element literals +// (`A[] & B[]`), and their arrays, tuples, and maps share one representation. + +interface Options { + root: string; + files?: Array<{ path: string; size: number }>; +} +type Normalized = Options & { files: Array<{ path: string; size: number }> }; + +function normalize(options: Options): Normalized { + return { ...options, files: options.files ?? [] }; +} + +function totalSize(n: Normalized): number { + let total = 0; + for (const f of n.files) total += f.size; + return total; +} + +const plain = normalize({ root: "/a" }); +console.log(plain.root, plain.files.length, totalSize(plain)); +const full = normalize({ + root: "/b", + files: [ + { path: "x.txt", size: 3 }, + { path: "y.txt", size: 4 }, + ], +}); +full.files.push({ path: "z.txt", size: 5 }); +console.log(full.files.map((f) => f.path).join(","), totalSize(full)); +console.log(JSON.stringify(full)); + +let maybe: Normalized | undefined; +if (process.argv.length < 99) maybe = { root: "/", files: [{ path: "a", size: 1 }] }; +console.log(maybe?.root, maybe?.files[0]?.path); + +// Nested records whose inner arrays are restated. +interface Settings { + name: string; + meta?: { tags: Array<{ key: string; value: string }> }; +} +type Resolved = Settings & { meta: { tags: Array<{ key: string; value: string }> } }; +const resolved: Resolved = { name: "s", meta: { tags: [{ key: "k", value: "v" }] } }; +resolved.meta.tags.push({ key: "k2", value: "v2" }); +for (const t of resolved.meta.tags) console.log(`${t.key}=${t.value}`); + +// Tuples, nested arrays, and maps. +interface Layout { + pair?: [{ x: number }, string]; + grid?: Array>; + index?: Map; +} +type Filled = Layout & { + pair: [{ x: number }, string]; + grid: Array>; + index: Map; +}; +const filled: Filled = { + pair: [{ x: 7 }, "seven"], + grid: [[{ cell: 1 }, { cell: 2 }], [{ cell: 3 }]], + index: new Map([["home", { hits: 2 }]]), +}; +filled.index.set("about", { hits: 1 }); +console.log(filled.pair[0].x, filled.pair[1]); +console.log(filled.grid.map((row) => row.map((c) => c.cell).join("+")).join(" | ")); +for (const [k, v] of filled.index) console.log(k, v.hits); + +// Both sides required, and both sides optional (the member stays optional). +type Twice = { items: Array<{ n: number }> } & { items: Array<{ n: number }> }; +const twice: Twice = { items: [{ n: 1 }, { n: 2 }] }; +console.log(twice.items.reduce((s, i) => s + i.n, 0)); +type BothOptional = { extra?: Array<{ n: number }> } & { extra?: Array<{ n: number }> }; +const none: BothOptional = {}; +const some: BothOptional = { extra: [{ n: 9 }] }; +console.log(none.extra === undefined, some.extra?.[0]?.n); + +// A restated member read through a function value. +type Handler = (event: Normalized) => string; +const describe: Handler = (event) => `${event.root}:${event.files.length}`; +console.log(describe(full), describe(plain)); diff --git a/tests/corpus/native-array-dynamic-reads/main.ts b/tests/corpus/native-array-dynamic-reads/main.ts new file mode 100644 index 000000000..0641114bf --- /dev/null +++ b/tests/corpus/native-array-dynamic-reads/main.ts @@ -0,0 +1,32 @@ +// Native arrays and iterators from typed code, read by untyped code: for-of +// over them, their length, an iterator's next read as a value and called, +// generators and async generators beside them, and an array whose +// iteration a Symbol.iterator use elsewhere in the program does not change. +import { asyncKinds, describe, lengths, sumAll, stepper } from "./reads.js"; + +interface Item { + sku: string; + qty: number; +} +const numbers: number[] = [1, 2, 3, 4]; +const items: Item[] = [ + { sku: "a", qty: 2 }, + { sku: "b", qty: 5 }, +]; +const empty: string[] = []; +function* counting(): Generator { + yield 10; + yield 20; +} +console.log(sumAll(numbers), sumAll(items.map((i) => i.qty)), sumAll(empty.map(Number))); +console.log(lengths([numbers, items, empty]).join()); +numbers.push(5); +console.log(lengths([numbers]).join(), sumAll(numbers)); +console.log(describe(items), describe(counting())); +console.log(stepper(numbers).join(), stepper(counting()).join()); +const iterable = { *[Symbol.iterator]() { yield "own"; } }; +console.log([...iterable].join(), sumAll(numbers)); +async function* later(): AsyncGenerator { + yield 1; +} +asyncKinds(later(), numbers).then((kinds) => console.log(kinds.join())); diff --git a/tests/corpus/native-array-dynamic-reads/reads.js b/tests/corpus/native-array-dynamic-reads/reads.js new file mode 100644 index 000000000..67e1a824a --- /dev/null +++ b/tests/corpus/native-array-dynamic-reads/reads.js @@ -0,0 +1,29 @@ +export function sumAll(values) { + let total = 0; + for (const value of values) total += value; + return total; +} +export function lengths(arrays) { + const out = []; + for (const array of arrays) out.push(array.length); + return out; +} +export function describe(values) { + const parts = []; + for (const value of values) parts.push(typeof value === "object" ? value.sku : value); + return parts.join("+"); +} +export function stepper(values) { + const iterator = values[Symbol.iterator](); + const next = iterator.next; + const out = []; + for (let step = next.call(iterator); !step.done; step = next.call(iterator)) out.push(step.value); + out.push(typeof iterator.next, iterator.next === next); + return out; +} +export async function asyncKinds(generator, array) { + const out = []; + for await (const value of generator) out.push(value); + for (const value of array) out.push(value); + return out; +} diff --git a/tests/corpus/native-worker-length-loop-reassign.ts b/tests/corpus/native-worker-length-loop-reassign.ts new file mode 100644 index 000000000..7ec152b16 --- /dev/null +++ b/tests/corpus/native-worker-length-loop-reassign.ts @@ -0,0 +1,58 @@ +// Loops to a typed array's length in a program that starts workers, where +// the body assigns the loop's array: the condition reads the new array's +// length on the next iteration, shorter or longer. +import { Worker, isMainThread, parentPort } from "node:worker_threads"; + +function shrink(values: Float64Array): string { + let total = 0; + let steps = 0; + for (let i = 0; i < values.length; i++) { + total += values[i]!; + steps++; + if (i === 100) values = values.subarray(0, 50); + } + return `shrink ${steps} ${total}`; +} + +function grow(values: Uint8Array, longer: Uint8Array): string { + let total = 0; + let steps = 0; + for (let i = 0; i < values.length; i++) { + total += values[i]!; + steps++; + if (i === 10) values = longer; + } + return `grow ${steps} ${total}`; +} + +function swap(a: Int32Array, b: Int32Array): string { + let total = 0; + let steps = 0; + for (let i = 0; i < a.length; i++) { + total += a[i]!; + steps++; + if (i % 70 === 69) { + const t = a; + a = b; + b = t; + } + } + return `swap ${steps} ${total}`; +} + +function run(): string[] { + const values = new Float64Array(300).map((_, i) => i * 0.5); + const short = new Uint8Array(20).fill(3); + const long = new Uint8Array(200).map((_, i) => i & 7); + const a = new Int32Array(100).map((_, i) => i); + const b = new Int32Array(160).map((_, i) => -i); + return [shrink(values), grow(short, long), swap(a, b)]; +} + +if (isMainThread) { + for (const line of run()) console.log(line); + const worker = new Worker(new URL(import.meta.url)); + worker.on("message", (lines: string[]) => { + for (const line of lines) console.log("worker", line); + }); +} else parentPort!.postMessage(run()); diff --git a/tests/corpus/native-worker-terminate-counted-loops.ts b/tests/corpus/native-worker-terminate-counted-loops.ts new file mode 100644 index 000000000..55972c4f6 --- /dev/null +++ b/tests/corpus/native-worker-terminate-counted-loops.ts @@ -0,0 +1,66 @@ +// worker.terminate() stops a worker in counted loops over a typed array +// with long bounds: ascending and descending counters, a step above one, a +// second counter that steps with the first, and loops to a typed array's +// length run again and again. +import { Worker, isMainThread, parentPort, workerData } from "node:worker_threads"; + +function ascending(values: Float64Array, limit: number): number { + let acc = 0; + for (let i = 0; i < limit; i++) { + acc = (acc + values[i & 1023]!) % 1000003; + if (i === 5000) parentPort!.postMessage("ready"); + } + return acc; +} + +function descending(values: Float64Array, limit: number): number { + let acc = 0; + for (let i = limit; i > 0; i -= 3) { + acc = (acc * 31 + i + values[i & 1023]!) % 1000003; + if (i === limit - 15000) parentPort!.postMessage("ready"); + } + return acc; +} + +function paired(values: Float64Array, limit: number): number { + let acc = 0; + for (let i = 0, j = 7; i < limit; i++, j += 2) { + acc = (acc + i * j + values[j & 1023]!) % 1000003; + if (i === 5000) parentPort!.postMessage("ready"); + } + return acc; +} + +function lengths(bytes: Uint8Array): number { + let acc = 0; + for (let round = 0; ; round++) { + for (let i = 0; i < bytes.length; ++i) bytes[i] = bytes[i] === 97 ? 0 : (bytes[i]! + round) & 127; + for (let i = 0; i < bytes.length; i++) acc = (acc + bytes[i]!) % 1000003; + if (round === 50) parentPort!.postMessage("ready"); + } + return acc; +} + +if (isMainThread) { + for (const kind of ["ascending", "descending", "paired", "lengths"]) { + await new Promise((resolve) => { + const worker = new Worker(new URL(import.meta.url), { workerData: kind }); + worker.once("message", () => { + void worker.terminate().then((code: number) => console.log(kind, "terminate", code)); + }); + worker.once("exit", (code: number) => { + console.log(kind, "exit", code); + resolve(); + }); + }); + } +} else { + const values = new Float64Array(1024).map((_, i) => i); + const limit = 1e15; + let result = 0; + if (workerData === "ascending") result = ascending(values, limit); + else if (workerData === "descending") result = descending(values, limit); + else if (workerData === "paired") result = paired(values, limit); + else result = lengths(new Uint8Array(1 << 16).fill(97)); + console.log("unexpected result", result); +} diff --git a/tests/corpus/typed-array-length-after-release.ts b/tests/corpus/typed-array-length-after-release.ts new file mode 100644 index 000000000..81fad4c77 --- /dev/null +++ b/tests/corpus/typed-array-length-after-release.ts @@ -0,0 +1,56 @@ +// A typed array's length and byteLength read just before the view's last +// reference goes away: the values stay those of the old view after a field +// replaces it, a map entry is deleted, or a local is rebound. +function grow(slots: Int32Array): Int32Array { + const next = new Int32Array(slots.length * 2); + next.set(slots); + return next; +} + +class Table { + slots: Int32Array = new Int32Array(4); + count = 0; + add(value: number): number { + const slots = this.slots; + slots[this.count % slots.length] = value; + this.count++; + if (this.count === slots.length) this.slots = grow(slots); + return this.slots.length; + } + swap(): number { + const slots = this.slots; + this.slots = new Int32Array(1); + return slots.length; + } + swapBytes(): number { + const slots = this.slots; + const bytes = slots.byteLength; + this.slots = new Int32Array(2); + return bytes + slots.length; + } +} + +const table = new Table(); +let size = 0; +for (let i = 0; i < 9; i++) size = table.add(i); +console.log(size, table.swap(), table.slots.length, table.add(5)); +console.log(table.swapBytes(), table.slots.length); + +const views = new Map(); +views.set("a", new Float64Array(6)); +function take(key: string): number { + const view = views.get(key); + if (view === undefined) return -1; + const length = view.length; + views.delete(key); + return length + view.byteLength; +} +console.log(take("a"), take("a"), views.size); + +let current = new Uint8Array(3); +function rebind(): number { + const old = current; + current = new Uint8Array(old.length + 1); + return old.length * 10 + current.length; +} +console.log(rebind(), rebind(), current.length); diff --git a/tests/fixtures/commander-calc/package.json b/tests/fixtures/commander-calc/package.json index f0ea76992..59f181b32 100644 --- a/tests/fixtures/commander-calc/package.json +++ b/tests/fixtures/commander-calc/package.json @@ -4,5 +4,15 @@ "type": "module", "dependencies": { "commander": "15.0.0" + }, + "scriptc": { + "typePatches": { + "commander@^15.0.0": { + "note": "off a TTY, _getOutputContext() returns helpWidth undefined although the @typedef says number", + "declarations": { + "HelpContext.helpWidth": "| undefined" + } + } + } } } diff --git a/tests/fixtures/npm/cases/type-patch-dts/main.ts b/tests/fixtures/npm/cases/type-patch-dts/main.ts new file mode 100644 index 000000000..1e442054b --- /dev/null +++ b/tests/fixtures/npm/cases/type-patch-dts/main.ts @@ -0,0 +1,15 @@ +// A package whose .d.ts says `email: string` for a value that is missing, +// patched to `string | undefined` (package.json scriptc.typePatches): the +// profiles cross the declaration boundary as native records, and user +// code handles the missing email the patched type now makes visible. +import { findProfile, loadProfiles } from "profilekit"; + +const profiles = loadProfiles(); +for (const p of profiles) console.log(p.id, p.name, p.email ?? "(no email)", "email" in p); +console.log(JSON.stringify(profiles)); +const emails = profiles.filter((p) => p.email !== undefined).map((p) => p.email!.toUpperCase()); +console.log(emails.join(",")); +const found = findProfile(3); +console.log(found === undefined ? "none" : found.name); +const missing = findProfile(9); +console.log(missing === undefined ? "none" : missing.name); diff --git a/tests/fixtures/npm/cases/type-patch-dts/node_modules/profilekit/index.d.ts b/tests/fixtures/npm/cases/type-patch-dts/node_modules/profilekit/index.d.ts new file mode 100644 index 000000000..445d78ea2 --- /dev/null +++ b/tests/fixtures/npm/cases/type-patch-dts/node_modules/profilekit/index.d.ts @@ -0,0 +1,10 @@ +// The declarations say every profile has an email and every lookup finds +// one; the JavaScript leaves email out for guests and returns undefined +// for unknown ids. +export type Profile = { + id: number; + name: string; + email: string; +}; +export declare function loadProfiles(): Profile[]; +export declare function findProfile(id: number): Profile; diff --git a/tests/fixtures/npm/cases/type-patch-dts/node_modules/profilekit/index.js b/tests/fixtures/npm/cases/type-patch-dts/node_modules/profilekit/index.js new file mode 100644 index 000000000..a20a082f6 --- /dev/null +++ b/tests/fixtures/npm/cases/type-patch-dts/node_modules/profilekit/index.js @@ -0,0 +1,14 @@ +"use strict"; +function row(id, name, email) { + const profile = { id: id, name: name }; + if (email !== null) profile.email = email; + return profile; +} +function loadProfiles() { + return [row(1, "ada", "ada@example.com"), row(2, "guest", null), row(3, "lin", "lin@example.com")]; +} +function findProfile(id) { + return loadProfiles().find((p) => p.id === id); +} +exports.loadProfiles = loadProfiles; +exports.findProfile = findProfile; diff --git a/tests/fixtures/npm/cases/type-patch-dts/node_modules/profilekit/package.json b/tests/fixtures/npm/cases/type-patch-dts/node_modules/profilekit/package.json new file mode 100644 index 000000000..9982f9eef --- /dev/null +++ b/tests/fixtures/npm/cases/type-patch-dts/node_modules/profilekit/package.json @@ -0,0 +1 @@ +{ "name": "profilekit", "version": "3.1.0", "main": "index.js", "types": "index.d.ts" } diff --git a/tests/fixtures/npm/cases/type-patch-dts/package.json b/tests/fixtures/npm/cases/type-patch-dts/package.json new file mode 100644 index 000000000..822491420 --- /dev/null +++ b/tests/fixtures/npm/cases/type-patch-dts/package.json @@ -0,0 +1,15 @@ +{ + "name": "type-patch-dts-case", + "private": true, + "type": "module", + "scriptc": { + "typePatches": { + "profilekit@^3.0.0": { + "declarations": { + "Profile.email": "| undefined", + "findProfile()": "| undefined" + } + } + } + } +} diff --git a/tests/fixtures/npm/cases/type-patch-types-twin/main.ts b/tests/fixtures/npm/cases/type-patch-types-twin/main.ts new file mode 100644 index 000000000..702b51972 --- /dev/null +++ b/tests/fixtures/npm/cases/type-patch-types-twin/main.ts @@ -0,0 +1,11 @@ +// A plain-JavaScript package typed by an @types twin that says `max: +// number` and `mean: number`; the project's type patches (keyed by the +// runtime package and its installed version) make them `number | +// undefined` and `number | null`. +import { totals } from "tallyjs"; + +for (const values of [[3, 9, 4], []]) { + const t = totals(values); + console.log(t.count, t.sum, t.max ?? "no max", t.mean === null ? "no mean" : t.mean.toFixed(2)); + console.log(JSON.stringify(t), Object.keys(t).join("|")); +} diff --git a/tests/fixtures/npm/cases/type-patch-types-twin/node_modules/@types/tallyjs/index.d.ts b/tests/fixtures/npm/cases/type-patch-types-twin/node_modules/@types/tallyjs/index.d.ts new file mode 100644 index 000000000..4a7b81dcc --- /dev/null +++ b/tests/fixtures/npm/cases/type-patch-types-twin/node_modules/@types/tallyjs/index.d.ts @@ -0,0 +1,7 @@ +export type Totals = { + count: number; + sum: number; + max: number; + mean: number; +}; +export declare function totals(values: number[]): Totals; diff --git a/tests/fixtures/npm/cases/type-patch-types-twin/node_modules/@types/tallyjs/package.json b/tests/fixtures/npm/cases/type-patch-types-twin/node_modules/@types/tallyjs/package.json new file mode 100644 index 000000000..fed3cc37a --- /dev/null +++ b/tests/fixtures/npm/cases/type-patch-types-twin/node_modules/@types/tallyjs/package.json @@ -0,0 +1 @@ +{ "name": "@types/tallyjs", "version": "0.4.0", "types": "index.d.ts" } diff --git a/tests/fixtures/npm/cases/type-patch-types-twin/node_modules/tallyjs/index.js b/tests/fixtures/npm/cases/type-patch-types-twin/node_modules/tallyjs/index.js new file mode 100644 index 000000000..88d5ffc8f --- /dev/null +++ b/tests/fixtures/npm/cases/type-patch-types-twin/node_modules/tallyjs/index.js @@ -0,0 +1,11 @@ +"use strict"; +// Plain JavaScript; @types/tallyjs describes it, except that an empty +// list has no max (undefined) and no mean (null). +function totals(values) { + var sum = 0; + for (var i = 0; i < values.length; i++) sum += values[i]; + var max = values.length === 0 ? undefined : Math.max.apply(null, values); + var mean = values.length === 0 ? null : sum / values.length; + return { count: values.length, sum: sum, max: max, mean: mean }; +} +exports.totals = totals; diff --git a/tests/fixtures/npm/cases/type-patch-types-twin/node_modules/tallyjs/package.json b/tests/fixtures/npm/cases/type-patch-types-twin/node_modules/tallyjs/package.json new file mode 100644 index 000000000..ad80ad430 --- /dev/null +++ b/tests/fixtures/npm/cases/type-patch-types-twin/node_modules/tallyjs/package.json @@ -0,0 +1 @@ +{ "name": "tallyjs", "version": "0.4.2", "main": "index.js" } diff --git a/tests/fixtures/npm/cases/type-patch-types-twin/package.json b/tests/fixtures/npm/cases/type-patch-types-twin/package.json new file mode 100644 index 000000000..53ec04ede --- /dev/null +++ b/tests/fixtures/npm/cases/type-patch-types-twin/package.json @@ -0,0 +1,15 @@ +{ + "name": "type-patch-types-twin-case", + "private": true, + "type": "module", + "scriptc": { + "typePatches": { + "tallyjs@~0.4.0": { + "declarations": { + "Totals.max": "| undefined", + "Totals.mean": "number | null" + } + } + } + } +} diff --git a/tests/fixtures/self-hosting/llvm-layouts.ts b/tests/fixtures/self-hosting/llvm-layouts.ts index 36d55bddf..8e19896d2 100644 --- a/tests/fixtures/self-hosting/llvm-layouts.ts +++ b/tests/fixtures/self-hosting/llvm-layouts.ts @@ -51,6 +51,7 @@ try { declare: (decl) => { declarations.add(decl); }, needOom: () => { needsOom = true; }, needBadTag: () => { needsBadTag = true; }, + mayHoldForeign: () => false, internLiteral: (text) => { let index = literals.indexOf(text); if (index === -1) { index = literals.length; literals.push(text); } diff --git a/tests/fixtures/type-patches/jsdoc.ts b/tests/fixtures/type-patches/jsdoc.ts new file mode 100644 index 000000000..52cd13638 --- /dev/null +++ b/tests/fixtures/type-patches/jsdoc.ts @@ -0,0 +1,11 @@ +// A JSDoc @typedef property and a @returns that say number for a value +// that is undefined off a TTY, fixed by the project's type patches. +import { Terminal } from "termctx"; + +for (const tty of [false, true]) { + const term = new Terminal(tty); + const context = term.outputContext(false); + console.log(context.error, context.width, term.columns()); + console.log(term.describe(context)); + console.log(term.describe(term.outputContext(true))); +} diff --git a/tests/fixtures/type-patches/ms.ts b/tests/fixtures/type-patches/ms.ts new file mode 100644 index 000000000..a70e221d6 --- /dev/null +++ b/tests/fixtures/type-patches/ms.ts @@ -0,0 +1,9 @@ +// ms's JSDoc says parse() returns a Number and ms() a String or Number; +// for text it cannot parse both return undefined. +import ms from "ms"; + +for (const text of ["2 hours", "not a duration", "1".repeat(101), "-1.5d"]) { + const value = ms(text); + console.log(value === undefined ? "invalid" : value); +} +console.log(ms(90_000), ms(90_000, { long: true })); diff --git a/tests/fixtures/type-patches/node_modules/ms/index.js b/tests/fixtures/type-patches/node_modules/ms/index.js new file mode 100644 index 000000000..ea734fb73 --- /dev/null +++ b/tests/fixtures/type-patches/node_modules/ms/index.js @@ -0,0 +1,162 @@ +/** + * Helpers. + */ + +var s = 1000; +var m = s * 60; +var h = m * 60; +var d = h * 24; +var w = d * 7; +var y = d * 365.25; + +/** + * Parse or format the given `val`. + * + * Options: + * + * - `long` verbose formatting [false] + * + * @param {String|Number} val + * @param {Object} [options] + * @throws {Error} throw an error if val is not a non-empty string or a number + * @return {String|Number} + * @api public + */ + +module.exports = function (val, options) { + options = options || {}; + var type = typeof val; + if (type === 'string' && val.length > 0) { + return parse(val); + } else if (type === 'number' && isFinite(val)) { + return options.long ? fmtLong(val) : fmtShort(val); + } + throw new Error( + 'val is not a non-empty string or a valid number. val=' + + JSON.stringify(val) + ); +}; + +/** + * Parse the given `str` and return milliseconds. + * + * @param {String} str + * @return {Number} + * @api private + */ + +function parse(str) { + str = String(str); + if (str.length > 100) { + return; + } + var match = /^(-?(?:\d+)?\.?\d+) *(milliseconds?|msecs?|ms|seconds?|secs?|s|minutes?|mins?|m|hours?|hrs?|h|days?|d|weeks?|w|years?|yrs?|y)?$/i.exec( + str + ); + if (!match) { + return; + } + var n = parseFloat(match[1]); + var type = (match[2] || 'ms').toLowerCase(); + switch (type) { + case 'years': + case 'year': + case 'yrs': + case 'yr': + case 'y': + return n * y; + case 'weeks': + case 'week': + case 'w': + return n * w; + case 'days': + case 'day': + case 'd': + return n * d; + case 'hours': + case 'hour': + case 'hrs': + case 'hr': + case 'h': + return n * h; + case 'minutes': + case 'minute': + case 'mins': + case 'min': + case 'm': + return n * m; + case 'seconds': + case 'second': + case 'secs': + case 'sec': + case 's': + return n * s; + case 'milliseconds': + case 'millisecond': + case 'msecs': + case 'msec': + case 'ms': + return n; + default: + return undefined; + } +} + +/** + * Short format for `ms`. + * + * @param {Number} ms + * @return {String} + * @api private + */ + +function fmtShort(ms) { + var msAbs = Math.abs(ms); + if (msAbs >= d) { + return Math.round(ms / d) + 'd'; + } + if (msAbs >= h) { + return Math.round(ms / h) + 'h'; + } + if (msAbs >= m) { + return Math.round(ms / m) + 'm'; + } + if (msAbs >= s) { + return Math.round(ms / s) + 's'; + } + return ms + 'ms'; +} + +/** + * Long format for `ms`. + * + * @param {Number} ms + * @return {String} + * @api private + */ + +function fmtLong(ms) { + var msAbs = Math.abs(ms); + if (msAbs >= d) { + return plural(ms, msAbs, d, 'day'); + } + if (msAbs >= h) { + return plural(ms, msAbs, h, 'hour'); + } + if (msAbs >= m) { + return plural(ms, msAbs, m, 'minute'); + } + if (msAbs >= s) { + return plural(ms, msAbs, s, 'second'); + } + return ms + ' ms'; +} + +/** + * Pluralization helper. + */ + +function plural(ms, msAbs, n, name) { + var isPlural = msAbs >= n * 1.5; + return Math.round(ms / n) + ' ' + name + (isPlural ? 's' : ''); +} diff --git a/tests/fixtures/type-patches/node_modules/ms/license.md b/tests/fixtures/type-patches/node_modules/ms/license.md new file mode 100644 index 000000000..fa5d39b62 --- /dev/null +++ b/tests/fixtures/type-patches/node_modules/ms/license.md @@ -0,0 +1,21 @@ +The MIT License (MIT) + +Copyright (c) 2020 Vercel, Inc. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/tests/fixtures/type-patches/node_modules/ms/package.json b/tests/fixtures/type-patches/node_modules/ms/package.json new file mode 100644 index 000000000..b49dca98d --- /dev/null +++ b/tests/fixtures/type-patches/node_modules/ms/package.json @@ -0,0 +1,38 @@ +{ + "name": "ms", + "version": "2.1.3", + "description": "Tiny millisecond conversion utility", + "repository": "vercel/ms", + "main": "./index", + "files": [ + "index.js" + ], + "scripts": { + "precommit": "lint-staged", + "lint": "eslint lib/* bin/*", + "test": "mocha tests.js" + }, + "eslintConfig": { + "extends": "eslint:recommended", + "env": { + "node": true, + "es6": true + } + }, + "lint-staged": { + "*.js": [ + "npm run lint", + "the formatter --single-quote --write", + "git add" + ] + }, + "license": "MIT", + "devDependencies": { + "eslint": "4.18.2", + "expect.js": "0.3.1", + "husky": "0.14.3", + "lint-staged": "5.0.0", + "mocha": "4.0.1", + "the formatter": "2.0.5" + } +} diff --git a/tests/fixtures/type-patches/node_modules/termctx/index.js b/tests/fixtures/type-patches/node_modules/termctx/index.js new file mode 100644 index 000000000..c8b2b9d2d --- /dev/null +++ b/tests/fixtures/type-patches/node_modules/termctx/index.js @@ -0,0 +1,43 @@ +"use strict"; +// A terminal helper whose JSDoc says widths are numbers; off a TTY they +// are undefined (the shape of commander's HelpContext.helpWidth). + +class Terminal { + constructor(tty) { + this.tty = tty; + this.settings = { + columns: () => (this.tty ? 80 : undefined), + }; + } + + /** + * @returns {number} + */ + columns() { + return this.settings.columns(); + } + + /** + * @typedef OutputContext + * @type {object} + * @property {boolean} error + * @property {number} width + * + * @param {boolean} error + * @returns {OutputContext} + */ + outputContext(error) { + const width = this.settings.columns(); + return { error, width }; + } + + /** + * @param {OutputContext} context + * @returns {string} + */ + describe(context) { + return `width ${context.width}${context.error ? " (stderr)" : ""}`; + } +} + +module.exports = { Terminal }; diff --git a/tests/fixtures/type-patches/node_modules/termctx/package.json b/tests/fixtures/type-patches/node_modules/termctx/package.json new file mode 100644 index 000000000..1791f57af --- /dev/null +++ b/tests/fixtures/type-patches/node_modules/termctx/package.json @@ -0,0 +1 @@ +{ "name": "termctx", "version": "1.2.0", "main": "index.js" } diff --git a/tests/fixtures/type-patches/package.json b/tests/fixtures/type-patches/package.json new file mode 100644 index 000000000..f1b9f630e --- /dev/null +++ b/tests/fixtures/type-patches/package.json @@ -0,0 +1,26 @@ +{ + "name": "type-patches-fixture", + "private": true, + "type": "module", + "dependencies": { + "ms": "2.1.3", + "termctx": "1.2.0" + }, + "scriptc": { + "typePatches": { + "termctx@^1.0.0": { + "declarations": { + "OutputContext.width": "| undefined", + "Terminal.columns()": "| undefined" + } + }, + "ms@^2.1.0": { + "note": "parse() returns undefined for text it cannot parse", + "declarations": { + "index.js:parse()": "| undefined", + "export=()": "| undefined" + } + } + } + } +} diff --git a/tests/fixtures/type-patches/stale/main.ts b/tests/fixtures/type-patches/stale/main.ts new file mode 100644 index 000000000..05ae25057 --- /dev/null +++ b/tests/fixtures/type-patches/stale/main.ts @@ -0,0 +1,4 @@ +import ms from "ms"; +import { Terminal } from "termctx"; + +console.log(ms(1000), new Terminal(true).columns()); diff --git a/tests/fixtures/type-patches/stale/package.json b/tests/fixtures/type-patches/stale/package.json new file mode 100644 index 000000000..53669c8b5 --- /dev/null +++ b/tests/fixtures/type-patches/stale/package.json @@ -0,0 +1,17 @@ +{ + "name": "type-patches-stale", + "private": true, + "type": "module", + "scriptc": { + "typePatches": { + "termctx@^2.0.0": { "declarations": { "OutputContext.width": "| undefined" } }, + "ms@^2.1.0": { + "declarations": { + "index.js:format()": "| undefined", + "index.js:fmtShort(ms)": "| Number", + "index.js:plural()": "| undefined" + } + } + } + } +} diff --git a/tests/harness/corpus-costs/6067-view-holes-in-typed-arrays.ts.json b/tests/harness/corpus-costs/6067-view-holes-in-typed-arrays.ts.json new file mode 100644 index 000000000..b8b740bf8 --- /dev/null +++ b/tests/harness/corpus-costs/6067-view-holes-in-typed-arrays.ts.json @@ -0,0 +1 @@ +{ "plain": 3967, "san": 4623 } diff --git a/tests/harness/corpus-costs/6068-foreign-elements-through-views.ts.json b/tests/harness/corpus-costs/6068-foreign-elements-through-views.ts.json new file mode 100644 index 000000000..eea1236ac --- /dev/null +++ b/tests/harness/corpus-costs/6068-foreign-elements-through-views.ts.json @@ -0,0 +1 @@ +{ "plain": 6595, "san": 5482 } diff --git a/tests/harness/corpus-costs/6069-foreign-element-writers.ts.json b/tests/harness/corpus-costs/6069-foreign-element-writers.ts.json new file mode 100644 index 000000000..48bc4a979 --- /dev/null +++ b/tests/harness/corpus-costs/6069-foreign-element-writers.ts.json @@ -0,0 +1 @@ +{ "plain": 3978, "san": 3898 } diff --git a/tests/harness/corpus-costs/7515-net-socket-backpressure.ts.json b/tests/harness/corpus-costs/7515-net-socket-backpressure.ts.json new file mode 100644 index 000000000..feabbe802 --- /dev/null +++ b/tests/harness/corpus-costs/7515-net-socket-backpressure.ts.json @@ -0,0 +1 @@ +{ "plain": 2800, "san": 4200 } diff --git a/tests/harness/corpus-costs/7519-net-socket-drain-listener-close.ts.json b/tests/harness/corpus-costs/7519-net-socket-drain-listener-close.ts.json new file mode 100644 index 000000000..78ee9421a --- /dev/null +++ b/tests/harness/corpus-costs/7519-net-socket-drain-listener-close.ts.json @@ -0,0 +1 @@ +{ "plain": 2300, "san": 3100 } diff --git a/tests/harness/corpus-costs/8010-http-header-raw-bytes.ts.json b/tests/harness/corpus-costs/8010-http-header-raw-bytes.ts.json new file mode 100644 index 000000000..532e3bd74 --- /dev/null +++ b/tests/harness/corpus-costs/8010-http-header-raw-bytes.ts.json @@ -0,0 +1 @@ +{ "plain": 3468, "san": 7463 } diff --git a/tests/harness/corpus-costs/8165-rsa-keygen-lengths.cjs.json b/tests/harness/corpus-costs/8165-rsa-keygen-lengths.cjs.json new file mode 100644 index 000000000..bbf64dca5 --- /dev/null +++ b/tests/harness/corpus-costs/8165-rsa-keygen-lengths.cjs.json @@ -0,0 +1 @@ +{ "plain": 11640, "san": 77506 } diff --git a/tests/harness/corpus-costs/8166-rsa-pss-keys.cjs.json b/tests/harness/corpus-costs/8166-rsa-pss-keys.cjs.json new file mode 100644 index 000000000..436e9ed51 --- /dev/null +++ b/tests/harness/corpus-costs/8166-rsa-pss-keys.cjs.json @@ -0,0 +1 @@ +{ "plain": 11459, "san": 74658 } diff --git a/tests/harness/corpus-costs/8167-encrypted-private-keys.cjs.json b/tests/harness/corpus-costs/8167-encrypted-private-keys.cjs.json new file mode 100644 index 000000000..0f258ddbb --- /dev/null +++ b/tests/harness/corpus-costs/8167-encrypted-private-keys.cjs.json @@ -0,0 +1 @@ +{ "plain": 3144, "san": 8504 } diff --git a/tests/harness/corpus-costs/8178-rsa-pss-auto-min-salt.cjs.json b/tests/harness/corpus-costs/8178-rsa-pss-auto-min-salt.cjs.json new file mode 100644 index 000000000..b1711dea1 --- /dev/null +++ b/tests/harness/corpus-costs/8178-rsa-pss-auto-min-salt.cjs.json @@ -0,0 +1 @@ +{ "plain": 5740, "san": 60808 } diff --git a/tests/harness/corpus-costs/8183-key-passphrase-encoding.cjs.json b/tests/harness/corpus-costs/8183-key-passphrase-encoding.cjs.json new file mode 100644 index 000000000..0d507484f --- /dev/null +++ b/tests/harness/corpus-costs/8183-key-passphrase-encoding.cjs.json @@ -0,0 +1 @@ +{ "plain": 3665, "san": 67315 } diff --git a/tests/harness/corpus-costs/8240-asserted-callee-errors.ts.json b/tests/harness/corpus-costs/8240-asserted-callee-errors.ts.json new file mode 100644 index 000000000..8c0e66c3c --- /dev/null +++ b/tests/harness/corpus-costs/8240-asserted-callee-errors.ts.json @@ -0,0 +1 @@ +{ "plain": 3000, "san": 3600 } diff --git a/tests/harness/corpus-costs/8347-string-iteration-code-points.ts.json b/tests/harness/corpus-costs/8347-string-iteration-code-points.ts.json new file mode 100644 index 000000000..3e2ae0706 --- /dev/null +++ b/tests/harness/corpus-costs/8347-string-iteration-code-points.ts.json @@ -0,0 +1 @@ +{ "plain": 2770, "san": 3750 } diff --git a/tests/harness/corpus-costs/8348-string-iteration-worker-characters.ts.json b/tests/harness/corpus-costs/8348-string-iteration-worker-characters.ts.json new file mode 100644 index 000000000..4c96af583 --- /dev/null +++ b/tests/harness/corpus-costs/8348-string-iteration-worker-characters.ts.json @@ -0,0 +1 @@ +{ "plain": 3960, "san": 3900 } diff --git a/tests/harness/corpus-costs/8351-regex-literal-replacers.ts.json b/tests/harness/corpus-costs/8351-regex-literal-replacers.ts.json new file mode 100644 index 000000000..56546d44c --- /dev/null +++ b/tests/harness/corpus-costs/8351-regex-literal-replacers.ts.json @@ -0,0 +1 @@ +{ "plain": 3074, "san": 44176 } diff --git a/tests/harness/corpus-costs/intersection-restated-members.ts.json b/tests/harness/corpus-costs/intersection-restated-members.ts.json new file mode 100644 index 000000000..1254aa6e4 --- /dev/null +++ b/tests/harness/corpus-costs/intersection-restated-members.ts.json @@ -0,0 +1 @@ +{ "plain": 4000 } diff --git a/tests/harness/corpus-costs/native-array-dynamic-reads/main.ts.json b/tests/harness/corpus-costs/native-array-dynamic-reads/main.ts.json new file mode 100644 index 000000000..ac2ef5ae8 --- /dev/null +++ b/tests/harness/corpus-costs/native-array-dynamic-reads/main.ts.json @@ -0,0 +1 @@ +{ "plain": 3567, "san": 6402 } diff --git a/tests/harness/corpus-costs/native-worker-length-loop-reassign.ts.json b/tests/harness/corpus-costs/native-worker-length-loop-reassign.ts.json new file mode 100644 index 000000000..5ee17db16 --- /dev/null +++ b/tests/harness/corpus-costs/native-worker-length-loop-reassign.ts.json @@ -0,0 +1 @@ +{ "plain": 3158, "san": 20404 } diff --git a/tests/harness/corpus-costs/native-worker-terminate-counted-loops.ts.json b/tests/harness/corpus-costs/native-worker-terminate-counted-loops.ts.json new file mode 100644 index 000000000..66a146275 --- /dev/null +++ b/tests/harness/corpus-costs/native-worker-terminate-counted-loops.ts.json @@ -0,0 +1 @@ +{ "plain": 3594, "san": 23593 } diff --git a/tests/harness/corpus-costs/typed-array-length-after-release.ts.json b/tests/harness/corpus-costs/typed-array-length-after-release.ts.json new file mode 100644 index 000000000..b062243b7 --- /dev/null +++ b/tests/harness/corpus-costs/typed-array-length-after-release.ts.json @@ -0,0 +1 @@ +{ "plain": 2800, "san": 2500 } diff --git a/tests/harness/errors.test.ts b/tests/harness/errors.test.ts index 5cd66a2f5..3e59e534f 100644 --- a/tests/harness/errors.test.ts +++ b/tests/harness/errors.test.ts @@ -870,6 +870,38 @@ console.log(\`\${register(7)}\`); }); }); +// An RSA-PSS key whose parameters restrict it to a digest the runtime does +// not sign with (sha512-224 and sha512-256, which OpenSSL reads) throws a +// coded SC2020 error naming the digest, instead of importing the key +// without the restrictions it carries. +describe("RSA-PSS keys restricted to unimplemented digests", () => { + test("importing one throws SC2020 naming the digest", async () => { + const r = await compileAndRun( + "pss-restricted-digest", + `import { createPublicKey } from "node:crypto"; +const keys = [ + "-----BEGIN PUBLIC KEY-----\\nMIGQMEEGCSqGSIb3DQEBCjA0oA8wDQYJYIZIAWUDBAIFBQChHDAaBgkqhkiG9w0B\\nAQgwDQYJYIZIAWUDBAIFBQCiAwIBCANLADBIAkEArMDQESqjeWDLqFa1E5SzsMl4\\n6tyUsEqcDKSwRUvPilv1AUudf/w6qaXlPnAxBmFjZYihdQSZlk0czRLOX3o7gwID\\nAQAB\\n-----END PUBLIC KEY-----\\n", + "-----BEGIN PUBLIC KEY-----\\nMIGQMEEGCSqGSIb3DQEBCjA0oA8wDQYJYIZIAWUDBAIBBQChHDAaBgkqhkiG9w0B\\nAQgwDQYJYIZIAWUDBAIGBQCiAwIBCANLADBIAkEAuOa9NkzXzMDelHCIEkf0mnKe\\nnQdalNUm9vZtIMwpDJcn5U0Vbhhd6MTfRPhSo5nW71Oh5eU7v5M3PMYOHS7cMQID\\nAQAB\\n-----END PUBLIC KEY-----\\n", +]; +for (const key of keys) { + try { + console.log(createPublicKey(key).asymmetricKeyType); + } catch (error) { + const e = error as Error & { code?: string }; + console.log(e.name, e.code, e.message); + } +} +`, + ); + expect(r.exitCode).toBe(0); + expect(r.stdout).toBe( + "Error SC2020 'RSA-PSS keys restricted to sha512-224' is part of the standard library types but has no scriptc lowering yet\n" + + "Error SC2020 'RSA-PSS keys restricted to sha512-256' is part of the standard library types but has no scriptc lowering yet\n", + ); + expect(r.stderr).toBe(""); + }); +}); + // A cipher name Node lists in crypto.getCiphers() but the runtime does not // implement throws a coded SC2020 error naming the cipher when the name is // computed; literal names are refused at compile time (node-types-fenced). diff --git a/tests/harness/string-iteration-workers.test.ts b/tests/harness/string-iteration-workers.test.ts new file mode 100644 index 000000000..bc1feca95 --- /dev/null +++ b/tests/harness/string-iteration-workers.test.ts @@ -0,0 +1,44 @@ +/* The one-character strings that string iteration shares between threads + * outlive the worker that first made them. On Linux a worker that exits + * without sharing anything returns its memory; with SCR_SA_DISCARD=poison + * that memory becomes inaccessible instead, so a cached character + * allocated from the worker's memory would fault when the main thread + * reads it. The corpus program pins Node.js parity; this suite runs it + * with the discard poisoned and immediate. Under SCRIPTC_SAN=1 the runtime + * keeps every worker's memory, so the check does not apply. */ +import { execFile } from "node:child_process"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { expect, test } from "vitest"; +import { compile } from "@scriptc/compiler"; + +const exec = promisify(execFile); +const program = join(import.meta.dirname, "../corpus/8348-string-iteration-worker-characters.ts"); +const sanitize = process.env["SCRIPTC_SAN"] === "1"; + +test.skipIf(process.platform !== "linux" || sanitize)( + "characters a discarded worker cached stay readable", + { timeout: 240_000 }, + async () => { + const outDir = await mkdtemp(join(tmpdir(), "scriptc-string-iteration-workers-")); + try { + const result = await compile(program, { + backend: "llvm", + outDir, + outPath: join(outDir, "program"), + }); + expect(result.ok, JSON.stringify(result.diagnostics)).toBe(true); + if (!result.ok) return; + const { stdout } = await exec(result.binaryPath, [], { + env: { ...process.env, SCR_SA_DISCARD: "poison", SCR_SA_DISCARD_DELAY_MS: "0" }, + timeout: 120_000, + }); + expect(stdout).toContain("main 1:61 1:e9 1:20 1:f1"); + expect(stdout).toContain("spread a|é| |ñ"); + } finally { + await rm(outDir, { recursive: true, force: true }); + } + }, +); diff --git a/tests/harness/type-patches.test.ts b/tests/harness/type-patches.test.ts new file mode 100644 index 000000000..ecd8ee923 --- /dev/null +++ b/tests/harness/type-patches.test.ts @@ -0,0 +1,143 @@ +/* Type patches (package.json scriptc.typePatches): project-owned fixes for + * npm declarations that do not describe what the package does. A JSDoc + * @typedef property and @returns tag that say number for an undefined + * value (the shape of commander's HelpContext.helpWidth), and ms's @return + * tags, compile natively and byte-match Node once patched; without the + * patches the same declarations throw where Node runs. Patches that do not + * apply as written warn (SC8030-SC8033). The `.d.ts` and `@types` cases + * run in npm.test.ts (type-patch-dts, type-patch-types-twin), and + * commander-calc carries the HelpContext.helpWidth patch. */ +import { execFile } from "node:child_process"; +import { cpSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { describe, expect, test } from "vitest"; +import { compile } from "@scriptc/compiler"; + +const execFileAsync = promisify(execFile); +const repoRoot = join(import.meta.dirname, "../.."); +const fixture = join(repoRoot, "tests/fixtures/type-patches"); +const outDir = join(repoRoot, "node_modules/.cache/scriptc-tests/type-patches"); +const sanitize = process.env["SCRIPTC_SAN"] === "1"; + +interface Run { + stdout: string; + stderr: string; + exitCode: number; +} + +async function run(cmd: string, args: string[]): Promise { + try { + const pending = execFileAsync(cmd, args, { encoding: "utf8" }); + pending.child.stdin?.end(); + const { stdout, stderr } = await pending; + return { stdout, stderr, exitCode: 0 }; + } catch (err) { + const e = err as { code?: unknown; stdout?: string; stderr?: string }; + if (typeof e.code !== "number") throw err; + return { stdout: e.stdout ?? "", stderr: e.stderr ?? "", exitCode: e.code }; + } +} + +async function build(entry: string, name: string, npmStatic: string[]) { + mkdirSync(outDir, { recursive: true }); + const result = await compile(entry, { outPath: join(outDir, name), outDir, sanitize, npmStatic }); + if (!result.ok) + throw new Error(result.diagnostics.map((d) => `${d.code}: ${d.message}`).join("\n")); + return result; +} + +/** A copy of a fixture directory whose package.json has no scriptc key. */ +function unpatched(dir: string): string { + const copy = mkdtempSync(join(tmpdir(), "scriptc-unpatched-")); + cpSync(dir, copy, { recursive: true }); + const manifest = join(copy, "package.json"); + const parsed = JSON.parse(readFileSync(manifest, "utf8")) as Record; + delete parsed["scriptc"]; + writeFileSync(manifest, JSON.stringify(parsed, null, 2)); + return copy; +} + +describe("type patches", () => { + test.for([ + ["jsdoc.ts", ["termctx"]], + ["ms.ts", ["ms"]], + ] as const)( + "%s compiles with its patches and byte-matches Node", + { timeout: 600_000 }, + async ([file, pkgs]) => { + const entry = join(fixture, file); + const result = await build(entry, file.replace(".ts", ""), [...pkgs]); + expect(result.warnings ?? []).toEqual([]); + const [node, native] = await Promise.all([ + run(process.execPath, [entry]), + run(result.binaryPath, []), + ]); + expect(node.exitCode).toBe(0); + expect(native).toEqual(node); + }, + ); + + test( + "without the patches the lying JSDoc throws where Node runs", + { timeout: 600_000 }, + async () => { + const copy = unpatched(fixture); + try { + const result = await build(join(copy, "jsdoc.ts"), "jsdoc-unpatched", ["termctx"]); + const native = await run(result.binaryPath, []); + expect(native.exitCode).not.toBe(0); + expect(native.stderr).toContain("TypeError"); + } finally { + rmSync(copy, { recursive: true, force: true }); + } + }, + ); + + test( + "without the patch the lying .d.ts throws at the declaration boundary", + { timeout: 600_000 }, + async () => { + const copy = unpatched(join(repoRoot, "tests/fixtures/npm/cases/type-patch-dts")); + try { + mkdirSync(outDir, { recursive: true }); + const result = await compile(join(copy, "main.ts"), { + outPath: join(outDir, "dts-unpatched"), + outDir, + sanitize, + }); + // Unpatched, `p.email ?? ...` and `found === undefined` test values + // the declarations say cannot be missing; the build may refuse them + // or the boundary cast throws when the guest profile crosses. + if (result.ok) { + const native = await run(result.binaryPath, []); + expect(native.exitCode).not.toBe(0); + expect(native.stderr).toContain("TypeError: expected string at $, got undefined"); + } + } finally { + rmSync(copy, { recursive: true, force: true }); + } + }, + ); + + test( + "patches that do not apply as written warn and the build goes on", + { timeout: 600_000 }, + async () => { + const entry = join(fixture, "stale/main.ts"); + const result = await build(entry, "stale", ["ms", "termctx"]); + expect((result.warnings ?? []).map((w) => w.code)).toEqual([ + "SC8030", + "SC8031", + "SC8032", + "SC8033", + ]); + const [node, native] = await Promise.all([ + run(process.execPath, [entry]), + run(result.binaryPath, []), + ]); + expect(native).toEqual(node); + }, + ); +});