readFileSync(path | fd, "utf8") (any ASCII-case utf8/utf-8 spelling and
the empty default) used to read into a Buffer, scan it for UTF-8
validity, then copy it into a new string while scanning it again for
ASCII. It now reads directly into the final string allocation, scans
each read chunk for ASCII while it is still in cache, and validates the
remainder in place (scr_str_adopt_utf8_lossy). Only malformed input is
repaired into a new string, with the same WHATWG maximal-subpart U+FFFD
replacement as Buffer#toString; a BOM is kept.
The raw-string readers behind fs.promises.readFile(path, "utf8") and
readFileSync(fd, "utf8") now decode the same way, so malformed bytes and
encoded surrogates become U+FFFD as in Node instead of passing through.
The ASCII prefix scan tests 32 bytes per step (scr_bytes_ascii_span),
which also speeds up Buffer and TextDecoder UTF-8 decoding of long ASCII
runs.
The corpus program covers empty files, BOMs, every malformed-subpart
class, truncation at EOF, chunk-boundary offsets, large ASCII and
multibyte files, /proc pseudo-files, fd and promise reads; the runtime
unit test checks the in-place decode under ASan and the RC audit.
Malte Ubl
b275f47fb3