mirror of
https://github.com/vercel-labs/scriptc.git
synced 2026-10-24 06:14:01 +08:00
Malte Ubl
20b7da29e7
The self-hosted compiler's compact IR is ~515-528M characters, a few million below V8's maximum string length (2^29 - 24). serializeModule joined it into one string, deserializeModule parsed one string, and the native bootstrap comparison read cli.ir.json and cli.ll as utf8 strings, so any compiler growth crashed --emit=ir and the bootstrap tests with "RangeError: Invalid string length". - serializeModuleParts returns the compact encoding as ordered parts (header, one part per function, remainder). serializeModule(mod, true) is their join, so the on-disk format is byte-identical. - deserializeModuleBytes splits a UTF-8 Buffer at the root object's "functions" array with a byte-level scanner and parses the header and each function separately. Malformed input falls back to the whole-document parse for its exact error. - The Node API writes --emit=ir, --emit-ir sidecars and --emit=llvm through writeTextParts (the former writeLlvmSource batching writer). - The native bootstrap harness decodes IR per function and compares LLVM as Buffers, removing ABI-check lines at the byte level. - The self-hosting serialization fixture gains a "bytes" mode so the scriptc-compiled serializer exercises both new entry points. Self-hosted compiler IR (514.6 MB, macOS arm64, Node 24.12): read old 5.85 s / 4.26 GB max RSS -> new 5.84 s / 1.36 GB write old 2.05-2.53 s / 4.48 GB -> new 2.30-2.37 s / 2.10 GB 2x module (1.03 GB): old serialize and read throw; new round-trips. The remaining per-string bound is the largest function (16.3M chars, ~33x headroom) and the header (2.3M chars).
32 lines
1.3 KiB
TypeScript
32 lines
1.3 KiB
TypeScript
/** Byte-level helpers for compiler-scale text artifacts (the self-hosted
|
|
* compiler's LLVM and IR). They avoid decoding a complete artifact into one
|
|
* JavaScript string, which V8 caps at 2^29 - 24 UTF-16 units. */
|
|
|
|
/** Offsets of every occurrence of `needle`. */
|
|
export function occurrences(bytes: Buffer, needle: string): number[] {
|
|
const found: number[] = [];
|
|
const width = Buffer.byteLength(needle);
|
|
for (let at = bytes.indexOf(needle); at >= 0; at = bytes.indexOf(needle, at + width))
|
|
found.push(at);
|
|
return found;
|
|
}
|
|
|
|
/** `bytes` without `length` bytes at `start`. */
|
|
export function withoutRange(bytes: Buffer, start: number, length: number): Buffer {
|
|
return Buffer.concat([bytes.subarray(0, start), bytes.subarray(start + length)]);
|
|
}
|
|
|
|
/** Remove the first complete line that starts with `prefix` and, including
|
|
* its newline, matches `line` (anchored), like `text.replace(/^...\n/m, "")`. */
|
|
export function withoutFirstLine(bytes: Buffer, prefix: string, line: RegExp): Buffer {
|
|
for (let from = 0; ;) {
|
|
const at = bytes.indexOf(prefix, from);
|
|
if (at < 0) return bytes;
|
|
const newline = bytes.indexOf(10, at);
|
|
const stop = newline < 0 ? bytes.length : newline + 1;
|
|
if ((at === 0 || bytes[at - 1] === 10) && line.test(bytes.toString("utf8", at, stop)))
|
|
return withoutRange(bytes, at, stop - at);
|
|
from = at + 1;
|
|
}
|
|
}
|