On x86-64 CPUs with the SHA extensions (and SSSE3 and SSE4.1), SHA-1 and
SHA-256 process their blocks with SHA1RNDS4/SHA256RNDS2 and the
message-schedule instructions, several blocks per call. The CPU check
runs once and is cached; other CPUs, other architectures and wasm32 keep
the portable blocks, and both give the same digests. createHash,
crypto.hash, createHmac, pbkdf2, hkdf and every other caller of the two
digests use the new path.
sha-kernels.test.ts runs both block implementations over known vectors
and every length from 0 to 1100 bytes and 1 MiB, whole and in 1-, 7-,
64- and 100-byte updates. Corpus 5763 compares the digests, HMAC, PBKDF2
and Hash.copy() with Node across block-boundary lengths and update
sizes.
Main now inlines dyn releases in compiled code (shapes.ts). Its fast path
buffered every surviving array and object, so compiled releases of a
parsed document's containers (the `parsed[i]` temporaries) bypassed the
intact-region check in scr_dyn_release. Arrays and objects now keep the
inline already-buffered check and otherwise call
scr_dyn_release_survivor, which skips containers of intact documents
(ScrDynRegion) like scr_dyn_release does. Functions and proxies keep the
inline enqueue.
Two writers that main added install tables or prototypes on existing
objects directly: scr_dyn_symbol_init_own and scr_dyn_dirent_finish. Both
now clear the object's region first, keeping the invariant that intact
containers have no symbol tables or prototypes.
scr_fss_new takes ownership of the FileHandle owner record (and its
reference to the handle), but its first two refusal paths returned before
the shared cleanup, so a FileHandle stream refused by its options, such
as fh.createReadStream({ encoding: "nope" }), leaked both. Those paths
now release the owner as the later ones do.
Corpus 5554 refuses a FileHandle stream and then closes the handle, and
tests/harness/fs-stream-leaks.test.ts runs the fs stream programs under
LeakSanitizer in the sanitized lane: 5554 reports the leak without this
change.
structuredClone of checked-dynamic values builds a fresh tree (shared
inputs are copied per path, cycles throw), so its containers join one
ScrDynRegion as they complete, exactly like JSON.parse; a BigInt or bytes
leaf clears the region, because intact containers promise JSON.stringify
inert leaves only.
scr_message_decode (worker, MessagePort and BroadcastChannel payloads)
tags its containers before dropping the decoder's references when every
node is a JSON primitive, undefined, array or object, arrays carry no
named properties, and every container edge leads to a higher encoder id.
Ids are assigned parents first, so that proves the graph acyclic; shared
subtrees reached again from a later node and cycles stay untagged.
Region API for other whole-document producers: scr_dyn_region_open,
scr_dyn_region_tag (on each completed container) and
scr_dyn_region_close. Node layout is unchanged: arrays and objects keep
the region pointer added for parse regions (arr.region / obj.region in
the capsule payload space).
The plain writer was inlined into scr_json_replaced_write, enlarging the
frame every level of the protocol walk recurses through. It is now out of
line and recurses into itself for member containers that prove plain, so
plain nesting costs one small frame per level and only members that are
not plain take the protocol walk. Pins a 900-level parsed document.
Numbers boxed by dyn expressions and the leaves of parsed documents are
most of the checked-dynamic churn. They own no dyn edges, are never cycle
candidates or weak keys, and never acquire a prototype or symbol table, so
their last release pushes them straight onto the misc pool and allocation
pops them with a minimal reset. Generic disposal now also clears the
released prototype, so every pooled node has NULL prototype and symbol
tables.
The dyn node pool kept at most 8192 nodes, while a parse/serialize loop
over a 100k-record document frees ~400k per round, so most nodes went back
through zeroing allocation and lost their preserved entry buffers. The cap
is now 1M nodes. The pool only keeps blocks the thread already allocated
(the small-object arena never returns them to the system either), at the
cost of reserving them, with their item/entry buffers, for dyn nodes.
Only JSON primitives and undefined now count as inert stores (published
immortal containers may hold functions, accessors or prototypes), so an
intact container provably holds only primitives and containers of its own
region. The plain stringify proof then needs no member inspection: O(1)
for arrays and a key scan for objects, instead of a pass that loaded every
member node (a cache miss per element) before the write loaded it again.
The plain JSON.stringify path proved only containers of primitive
members, so an array of parsed objects took the full protocol walk per
element: a formatted index key, a retained member read, a toJSON lookup
and the replacer callback. Inside an intact parse region, container
members are admitted too: everything such a container reaches was built
by the parser and holds JSON values only, so writing a member subtree
runs no program code that could mutate the container mid-walk and no
cycle passes through it. Each member container is still written through
scr_json_replaced_write, which proves it afresh.
Every heap dyn node carries a cycle header, so a member read of a parsed
document (parsed[i], JSON.stringify's walk) left each container it touched
in the candidate buffer and the collector traced the document over and
over. JSON.parse now tags the containers it builds with one shared region
that stays intact until the first store into any of them that could close
a cycle (a reference store; primitive and immortal stores, removals and
permutations keep it). Releases of intact containers skip buffering.
The flag is per region, not per container: a cycle can run through a
container that was never written (child.parent = root), and the last
outside reference to it may be dropped from that container. The write
barrier sits in the checked-dynamic writers (arr_push, obj_put, key_set,
define_property, reflect_define, set_prototype, symbol and named-property
tables, the reviver walk, unshift/splice/fill). A missed barrier would
leak a cycle, which the RC audit reports; walks and teardowns are
unchanged, so it cannot free live data.
Also corrects the collector header comment, which listed dyn trees as
headerless.
dgram.Socket's setBroadcast, setTTL, setMulticastTTL,
setMulticastLoopback, setMulticastInterface, addMembership,
dropMembership, get/setRecvBufferSize, get/setSendBufferSize,
getSendQueueSize, getSendQueueCount and disconnect were refused at
compile time. They now lower to scr_dgram.c and match Node:
- The TTL setters and setMulticastLoopback return their argument. The
value checks run first: a TTL outside 1-255 (0-255 for multicast,
after ToInt32) throws "setTTL EINVAL" with code, errno and syscall.
- A socket that has not been bound has no descriptor yet, so the
options throw EBADF, and the buffer size methods throw Node's
ERR_SOCKET_BUFFER_SIZE SystemError.
- After close(), the members that read Node's null handle throw its
TypeError ("Cannot read properties of null (reading 'setTTL')"), and
setMulticastInterface and the membership calls throw "Not running".
- Buffer sizes must be uint32 values (ERR_SOCKET_BAD_BUFFER_SIZE), 0
reads the current size, and sizes past the int32 range fail as EINVAL.
- IPv6 interfaces and groups go to the IPv6 socket options, so a udp4
socket answers ENOPROTOOPT as in Node.
- disconnect() throws ERR_SOCKET_DGRAM_NOT_CONNECTED until 'connect' was
emitted, and dissolves the association with an AF_UNSPEC connect.
addMembership and dropMembership before bind() throw SC2020, where
Node binds the socket to an ephemeral port first; the limitations page
documents it, along with the missing info object on the buffer size
SystemError. Corpus 5655 and 5656 and the node-types fixture
dgram-options.ts cover each method.
`"p" + n` and other string-plus-number concatenations whose left string
cannot grow in place (a literal or a shared string) went through the
general mixed concatenation, which sizes and copies every part through a
loop over a scratch table. The result is now the head's bytes followed by
the number's text in one exactly sized string. A number's text completes
no surrogate pair, so no seam needs joining. A uniquely owned head still
appends in place.
A const binding of promisify(exec) now lowers like promisify(execFile):
each call runs the command line with /bin/sh -c, as execSync does, and
settles with { stdout, stderr }. A non-zero exit, a missing command (the
shell's 127) or a timeout rejects with Node's "Command failed: <command>\n
<stderr>" message. The options are the execFile slice: encoding utf8,
cwd, env (inline spread included), timeout, and maxBuffer accepted.
Three gaps closed for both targets along the way:
- the { stdout, stderr } result maps as a record when it is held in a
variable, passed or returned, instead of only when destructured at the
await (it is an anonymous literal in @types/node and in the fallback);
- a namespace member is a recognized target (promisify(cp.exec));
- util.promisify(...) through a namespace or default import is
recognized like the named import.
A Map or Set header held six adapter pointers, the union key table and
size_t counts: 168 bytes, 176 in its size class. The adapters and union
key kinds now live in an immutable per-type ScrMapOps descriptor that the
compiler emits once per collection type, a traced byte replaces the two
trace-pointer tests, and the counts and capacities are 32-bit. The header
is 88 bytes.
The entries array starts with one slot and grows to four, then doubles,
and maps of up to 8 entries scan their cached hashes instead of keeping a
bucket table. Number-key maps still move to the direct index from four
entries.
One million maps held at once (RSS per map): empty 188 -> 100 bytes,
1 entry 286 -> 125, 4 entries 286 -> 198, 8 entries 450 -> 295.
Every release of a LifetimeRegion, and every retry at region creation, ran
scr_collect_cycles(): full cycle-collector passes over the thread's heap, to
a fixpoint, before checking any region, in case dead cycles of ordinary
objects held counted references into it. Freeing a region ran them again
whenever the owned ordinary objects left counts behind after the region
released its fields. Next to a large heap of candidates (a type checker
beside per-file emit regions) those passes cost far more than the regions.
A retry now checks the due regions first. A region whose references are all
explained frees without a pass; only when one would be kept does the heap
get the full pass, after which every due region is judged as before. The
first check leaves a region it cannot free as it was, so the backoff, the
kept report and SCRIPTC_REGION_DEBUG see only the check after the pass. It
keeps what its walk found: a region whose counted references the pass did not
change is judged from that, without walking it again. A pass that freed a dead
cycle referring only to an owned object leaves the count unchanged; such a
region waits for its next attempt.
After a region releases its instances' fields, what can still hold counts
is the owned ordinary objects' cycles. Each has a member that lost a
reference in those releases and is buffered as a candidate, and the release
walk entered every cycle member. The cycle collector gains a pass limited to
a set of candidates (scr_cyc_collect_set): it takes the set's members out of
the buffers up to a given generation, reading only the buffers' pointers, so
the candidates it leaves are not touched, and it walks at full depth without
moving the schedule's baselines. A dead object such a pass reaches through a
candidate it did not take leaves its buffer before it is freed. Freeing a
region runs these passes over the walk's objects from the youngest buffer
out, and the whole heap only if counts remain.
SCRIPTC_REGION_STATS also prints, per thread at exit, how many full passes
over the heap regions needed.
A collection pass copied every buffered candidate from the generation
root buffers into its candidate list, so it held two buffers of every
candidate, and each work vector kept its largest size for the rest of
the program.
A pass now takes the fullest root buffer as its candidate list and
appends the others; that generation gets the previous candidate buffer,
empty. The vectors a pass fills and leaves empty (pending, white,
promote, restore and cross-generation lists) record how many entries the
pass held at most. When a pass ends, any of them that holds more than
65536 entries is freed if the pass used less than a quarter of it, and so
is the candidate list's buffer when this pass's candidates filled less
than a quarter of it. Passes of a steady size keep reusing their
buffers; a spike in one pass no longer keeps its buffers for the rest of
the program.
When an older generation's buffer is the fullest, candidates are
visited, and a pass's garbage is freed, in a different order; the set
of objects a pass frees does not change.
JSON.parse appends each property and element, so object and array
storage grows by doubling and keeps up to half of its capacity unused,
and a parsed tree is usually kept as it is. When an object or array
closes, its storage now moves to a block of exactly its length (shrinking
in place would keep the block's size class). A table recycled from the
free list may hold keys past its length; they are released first.
Properties and elements added later grow the storage again as before.
Each object a publish walks cost two calls out of its emitted walker:
scr_pub_mark to mark and log it, and scr_pub_push for each reference it
holds (a string field went through scr_pub_push to scr_pub_str_fn).
The walkers now mark through ScrPub's leading fields: when the publish is
not refused, the log has room, and the count fits the log's 32 bits, they
write the log entry and SIZE_MAX themselves, and call scr_cyc_pub_take for
a buffered candidate. Anything else, and every mark of an RC-audit build,
still calls scr_pub_mark. References go onto the stack directly, with
scr_pub_push only when it is full, and string fields call scr_pub_str_fn.
ScrPub's fields are reordered so the walkers' fields come first; static
assertions in scr_publish.c pin the offsets the walkers use.
A plain socket with no 'data' listener, pipe or 'readable' consumer was
never read, so a peer that ended or closed the connection was never
noticed and the process stayed up. Node's stream reads ahead up to its
high-water mark (64 KiB) even without a consumer: a peer that ends
without sending data ends and closes the socket, unread bytes stay
buffered (bytesRead counts them) and keep the socket open, and a
listener added later receives them before 'end'. The runtime now does
the same, holding EOF back while unread bytes remain.
Corpus 5653 covers each case.
publish() logs each object it marks, with the object's count before the
mark, so that a refused publish can restore the graph. At 24 bytes per
object the log was most of the walk's memory traffic. Each publish also
allocated the log anew, so a large one faulted in fresh pages every time,
and a thread that publishes one graph per file paid that on each file.
A log entry is now 16 bytes on 64-bit targets. The count is stored in 32
bits, with escapes for the program-lifetime count and for any other count
of 32 bits or more, which goes to a side array restored in order on
refusal. The dyn-container flag moves into the entry's pointer, and the
RC audit's object kinds into a side array of audit builds. The stack and
log of a thread's last publish are kept for its next one while they take
16 MiB or less, and freed at thread exit.
Adds corpus program 5704.
When the main thread's script ends, the executable released every module
global and ran a full cycle collection before exiting, which takes time
proportional to everything the program still holds (about 15 ms for
1.5 million live objects) and frees memory the operating system reclaims
anyway. The main thread now skips those releases, the exit-time
collection and the collector's buffer cleanup. Exit listeners and output
flushing run as before. Worker threads still free their heaps when they
exit, since later threads reuse their arenas, and builds that check for
leaks (the RC audit, AddressSanitizer) free everything as before, as does
any process run with SCRIPTC_EXIT_FREE=1 for external leak checkers.
SCRIPTC_WORKER_STATS=1 reports, as each worker thread exits, how long it
took to start, to run its script and to exit.
Buffer.from of strings, arrays and Uint8Arrays, Buffer.allocUnsafe and
Buffer.concat now return views into a per-thread pool slab of
Buffer.poolSize bytes (64 KiB by default) when the size is below
Buffer.poolSize >>> 1, following Node's allocate, fromStringFast and
fromArrayLike: 8-byte aligned offsets, a new slab when a request does not
fit, and the four-bytes-per-unit reservation for strings. Buffer.poolSize
can be read and set, and Buffer.allocUnsafeSlow returns an unpooled Buffer.
A slab lives while any of its views does. Its ArrayBuffer cannot be
transferred, and structuredClone of a typed array view copies the whole
ArrayBuffer and keeps the view's offset. Buffer.alloc, allocUnsafe and
allocUnsafeSlow validate sizes with Node's ERR_OUT_OF_RANGE error.
The first pooled allocation finds the startup slab, which has the
default 64 KiB size. A request it cannot hold gets a slab of the current
Buffer.poolSize, as in Node, and the room is checked after every new
slab, so a Buffer.poolSize raised before the first pooled allocation
cannot place a reservation past the end of a 64 KiB slab. A reservation
larger than a fresh slab traps instead of writing past it.
Corpus 5531 (typed) and 5532 (untyped) raise Buffer.poolSize before the
first pooled allocation and allocate just under the new threshold
through each pooled path, each case in a fresh child process, and also
shrink Buffer.poolSize and change it between allocations. The
node-types fenced fixture no longer lists Buffer.poolSize, which now
compiles.
Checking or converting a dyn value whose objects or arrays refer back to
each other against a recursive interface (`raw.next = raw; raw as Node`)
recursed without end and overflowed the stack. Matchers and builders of
record and array types that reach themselves now track the dyn values
they visit:
- a matcher answers true for a (value, type) pair it is already checking
further up the stack, so the outer check decides;
- a builder records the record or array it allocated for a dyn value
before converting its members, and a later visit of the same value in
the conversion answers it, so `n.next === n` holds after the cast and a
value reached twice converts once.
Only values with more than one reference are tracked, since a value
reached twice has at least two; the reference a member read holds is not
counted. Capsules materialized during a conversion keep their first view,
so a nested visit cannot refresh a view an outer visit is reading, and
getters and proxy traps that a member read calls run outside the visit. The
cycle analysis treats record shapes that reach themselves and are targets
of a dynCheck as mutable, so the cyclic records a conversion builds are
collected.
Chris Tate