Refuse InternTable array keys that are too long to encode

Encoding an array key added each element's size to a running length
with unchecked arithmetic, and wrote into a fixed stack buffer whenever
the sum fit. A sparse array with a large length (2^29 elements or more)
wrapped the sum on 32-bit targets, so the write landed before the buffer;
on every target the encoder also visited every index of such an array.

Array keys now have at most 65536 elements, and their encoding at most
2^31 - 1 bytes. The size is computed first with sums that cannot wrap,
the buffer is sized to it, and keys over either limit make the lookup or
insert throw a TypeError before anything is read or written. The Node.js
implementation throws the same TypeError for keys of more than 65536
elements. Sparse elements are walked in one pass in index order, and the
index growth computation checks its size arithmetic too.
This commit is contained in:
Chris Tate
2026-10-15 21:44:38 -05:00
parent ee49946d32
commit 8dc5dc1249
13 changed files with 191 additions and 38 deletions
+1 -1
View File
@@ -159,7 +159,7 @@ worker: square: 4 lines, 3 shapes
<dl>
<dt><code>new InternTable&lt;K, V&gt;(name)</code></dt>
<dd>Opens the table called <code>name</code>. Within one thread every construction with the same name refers to the same table. In scriptc builds every thread of the process shares it; under Node.js each thread has its own. Keys are strings, numbers, or arrays of numbers or of strings. Strings and numbers compare like <code>Map</code> keys; arrays compare by their elements in the same way.</dd>
<dd>Opens the table called <code>name</code>. Within one thread every construction with the same name refers to the same table. In scriptc builds every thread of the process shares it; under Node.js each thread has its own. Keys are strings, numbers, or arrays of numbers or of strings. Strings and numbers compare like <code>Map</code> keys; arrays compare by their elements in the same way and can have at most 65536 elements.</dd>
<dt><code>table.get(key)</code>, <code>table.has(key)</code>, <code>table.size</code></dt>
<dd>Read the table.</dd>
<dt><code>table.getOrInsert(key, value)</code></dt>
@@ -8650,7 +8650,8 @@ export class LlEmitter {
/** InternTable's runtime calls (scr_itable.c). Table and entry handles
* are f64 values carrying pointer bits; entry values are 64-bit slots
* (double bits, a bool, or a pointer to an immortal value) that the
* table neither retains nor releases. */
* table neither retains nor releases. Lookups and inserts of array keys
* throw a TypeError for keys longer than a table accepts. */
private emitInternTableIntrinsic(e: ExprOf<"intrinsic">): LlValue {
const B = this.B;
// The runtime copies keys and stores published values without counts,
@@ -8692,6 +8693,8 @@ export class LlEmitter {
const call = keyCall("find");
const entry = B.tmp();
B.line(`${entry} = call ptr ${call})`);
// Array keys longer than a table accepts throw a TypeError.
if (e.args[1]!.type.kind === "array") this.emitPendingCheck();
return toHandle(entry);
}
case "itable.insert": {
@@ -8711,6 +8714,7 @@ export class LlEmitter {
const call = keyCall("insert");
const entry = B.tmp();
B.line(`${entry} = call ptr ${call}, i64 ${slot})`);
if (e.args[1]!.type.kind === "array") this.emitPendingCheck();
return toHandle(entry);
}
case "itable.value": {
@@ -196,6 +196,12 @@ export function computeMayThrow(
if (rec.name === "module.tdzCheck") f.throws = true;
// publish() throws on an unpublishable dyn value.
if (rec.name === "threads.publish") f.throws = true;
// An InternTable array key longer than a table accepts.
if (
(rec.name === "itable.find" || rec.name === "itable.insert") &&
rec.args[1]?.type.kind === "array"
)
f.throws = true;
break;
case "yieldExpr":
// A consumer .throw() surfaces at the yield (and .return()'s
+5 -3
View File
@@ -3482,9 +3482,11 @@ export type IrExpr =
* key's entry, storing the already published value when the key has
* none; itable.value (f64 entry) → V, the entry's immortal value;
* itable.size (f64 table) → f64. K is string, f64, or an array of
* f64 or of strings (compared by elements). Handles carry
* raw pointer bits and only reach toBool and these intrinsics.
* None of them throws. */
* f64 or of strings (compared by elements); find and insert throw
* a TypeError for array keys of more than 65536 elements or whose
* encoding exceeds 2^31 - 1 bytes; nothing else throws. Handles
* carry raw pointer bits and only reach toBool and these
* intrinsics. */
| "itable.open"
| "itable.find"
| "itable.insert"
@@ -0,0 +1,6 @@
{
"order": [
"<repo>/tests/corpus/5726-intern-table-key-limits.ts"
],
"diags": []
}
+1 -1
View File
@@ -55,7 +55,7 @@ function unionOf(members: readonly Type[]): Type {
```
- `new InternTable<K, V>(name)` opens the table called `name`. Within one thread, every construction with the same name refers to the same table. In scriptc builds the table is shared by every thread of the process; under Node.js each thread has its own.
- Keys are strings, numbers, or arrays of numbers or of strings. Strings and numbers compare like `Map` keys: strings by content, numbers by SameValueZero. Arrays compare by their elements in the same way, so a freshly built `[a.id, b.id]` finds the value stored under an equal array; holes are equal to each other and to no number or string.
- Keys are strings, numbers, or arrays of numbers or of strings. Strings and numbers compare like `Map` keys: strings by content, numbers by SameValueZero. Arrays compare by their elements in the same way, so a freshly built `[a.id, b.id]` finds the value stored under an equal array; holes are equal to each other and to no number or string. An array key can have at most 65536 elements; longer ones make every method throw a `TypeError`.
- `get(key)`, `has(key)` and `size` read the table.
- `getOrInsert(key, value)` returns the value stored under `key`. When there is none, it publishes `value`, stores it and returns it.
- `getOrInsertComputed(key, callback)` returns the value stored under `key`. When there is none, it calls `callback(key)`, publishes the result and stores it, unless a value was stored under `key` meanwhile (by the callback, or in scriptc builds by another thread), and returns the stored value.
+2 -1
View File
@@ -35,7 +35,8 @@ export declare class InternTable<
/** The value stored under `key`, or undefined. String and number keys
* compare like Map keys: strings by content, numbers by SameValueZero.
* Array keys compare by their elements in the same way; a hole and an
* undefined element are equal. */
* undefined element are equal. Every method throws a TypeError for an
* array key of more than 65536 elements. */
get(key: K): V | undefined;
/** Whether `key` has a value. */
has(key: K): boolean;
+4 -1
View File
@@ -24,9 +24,12 @@ const tables = new Map();
/** The Map key of a table key: strings and numbers as they are, arrays as a
* string that encodes each element (numbers by SameValueZero, strings by
* content, holes and undefined elements alike). */
* content, holes and undefined elements alike). Arrays of more than 65536
* elements are refused, as in scriptc builds. */
function keyOf(key) {
if (!Array.isArray(key)) return key;
if (key.length > 65536)
throw new TypeError("InternTable array keys can have at most 65536 elements");
let text = "";
for (let i = 0; i < key.length; i++) {
const element = key[i];
+84 -27
View File
@@ -10,7 +10,8 @@
* tag of hash bits 50..57 in bits 48..55, so a probe reads an entry only
* when the tags match. An array key is stored as one byte string that
* encodes its elements (scr_itable_encode), so array keys compare by
* their elements.
* their elements; keys longer or larger than a table accepts are refused
* with a TypeError before anything is encoded.
*
* Readers load the segment's index pointer, then slots with acquire order,
* and never write. Writers hold the segment's lock: they probe again, grow
@@ -44,6 +45,12 @@
#define SCR_ITABLE_FIRST_SLOTS 8u
#define SCR_ITABLE_FIRST_CHUNK ((size_t)512)
#define SCR_ITABLE_MAX_CHUNK ((size_t)64 << 10)
/* Keys whose bytes (a string's, or an array key's encoding) exceed this
* are refused, which keeps every entry size representable on 32-bit
* targets. Array keys of more than SCR_ITABLE_MAX_KEY_ELEMENTS elements are
* refused too; the Node.js implementation applies the same limit. */
#define SCR_ITABLE_MAX_KEY_BYTES ((size_t)SCR_STR_MAX_CAP)
#define SCR_ITABLE_MAX_KEY_ELEMENTS 65536u
/* String slots: 0 is empty; otherwise the entry pointer with the tag in
* bits 48..55, which no supported target uses for user-space addresses
@@ -314,7 +321,10 @@ const ScrITableEntry *scr_itable_find_f64(ScrITable *t, double key) {
/* ── inserts (the segment lock held) ─────────────────────────────────── */
static ScrITableIndex *scr_itable_index_new(uint64_t slots, bool numbers) {
size_t words = slots * (numbers ? 2 : 1) * sizeof(uint64_t);
/* Bytes per slot: the words and, for numbers, the tag byte. */
size_t per = (numbers ? 2 : 1) * sizeof(uint64_t) + (numbers ? 1 : 0);
if (slots > (SIZE_MAX - sizeof(ScrITableIndex)) / per) scr_itable_oom();
size_t words = (size_t)slots * (numbers ? 2 : 1) * sizeof(uint64_t);
ScrITableIndex *index = calloc(1, sizeof(ScrITableIndex) + words + (numbers ? slots : 0));
if (!index) scr_itable_oom();
index->mask = slots - 1;
@@ -394,7 +404,7 @@ static void *scr_itable_alloc(ScrITableSegment *seg, size_t size) {
* unless the key has an entry. */
static const ScrITableEntry *scr_itable_insert_bytes(ScrITable *t, uint64_t hash, const char *bytes,
size_t len, uint64_t value) {
if (len > UINT32_MAX) scr_trap("scriptc: InternTable key is too long\n");
if (len > SCR_ITABLE_MAX_KEY_BYTES) scr_trap("scriptc: InternTable key is too long\n");
ScrITableSegment *seg = scr_itable_segment(t, hash);
scr_itable_lock(&seg->lock);
const ScrITableStrEntry *found =
@@ -444,8 +454,16 @@ const ScrITableEntry *scr_itable_insert_f64(ScrITable *t, double key, uint64_t v
/* ── array keys ──────────────────────────────────────────────────────── */
/* The state and slot of element `i` of `a` (i < a->len). */
static ScrArrState scr_itable_element(const ScrArr *a, size_t i, uint64_t *slot) {
/* A walk over the elements of an array in index order. Sparse entries
* are sorted by index (scr_array.h), so one pass visits each once. */
typedef struct {
const ScrArr *a;
size_t sparse;
} ScrITableElements;
/* The state and slot of element `i` (i < len, increasing across calls). */
static ScrArrState scr_itable_element(ScrITableElements *w, size_t i, uint64_t *slot) {
const ScrArr *a = w->a;
if (i < a->filled) {
*slot = a->data[i];
return SCR_ARR_VALUE;
@@ -454,28 +472,65 @@ static ScrArrState scr_itable_element(const ScrArr *a, size_t i, uint64_t *slot)
*slot = a->data[i];
return (ScrArrState)a->present[i];
}
if (a->ext)
for (size_t j = 0; j < a->ext->sparse_len; j++)
if (a->ext->sparse[j].index == i) {
*slot = a->ext->sparse[j].slot;
return (ScrArrState)a->ext->sparse[j].state;
}
if (a->ext) {
while (w->sparse < a->ext->sparse_len && a->ext->sparse[w->sparse].index < i) w->sparse++;
if (w->sparse < a->ext->sparse_len && a->ext->sparse[w->sparse].index == i) {
*slot = a->ext->sparse[w->sparse].slot;
return (ScrArrState)a->ext->sparse[w->sparse].state;
}
}
return SCR_ARR_HOLE;
}
/* An array key as bytes: per number element its normalized bits, per
* string element a 32-bit length and the bytes; a hole or undefined
* element is the bits of a NaN that normalization never produces, or the
* length 0xffffffff. Answers the encoded length; writes at most `cap`
* bytes into `out`. */
* length 0xffffffff. */
#define SCR_ITABLE_ABSENT_F64 UINT64_C(0x7ff4000000000002)
#define SCR_ITABLE_ABSENT_STR UINT32_C(0xffffffff)
static size_t scr_itable_encode(const ScrArr *key, int kind, char *out, size_t cap) {
static void scr_itable_refuse(const char *message) {
scr_throw_error_msg(SCR_ERR_TYPE, message, strlen(message));
}
/* The encoded size of `key`, or false (with a pending TypeError) when the
* key is longer or larger than a table accepts. Every sum stays at most
* SCR_ITABLE_MAX_KEY_BYTES, so none can wrap. */
static bool scr_itable_encoded_size(const ScrArr *key, int kind, size_t *out) {
if (key->len > SCR_ITABLE_MAX_KEY_ELEMENTS) {
scr_itable_refuse("InternTable array keys can have at most 65536 elements");
return false;
}
size_t n = 0;
ScrITableElements walk = {key, 0};
for (size_t i = 0; i < key->len; i++) {
size_t add = kind == SCR_ITABLE_KEY_F64S ? 8 : 4;
if (kind != SCR_ITABLE_KEY_F64S) {
uint64_t slot = 0;
if (scr_itable_element(&walk, i, &slot) == SCR_ARR_VALUE) {
const ScrStr *s = (const ScrStr *)(uintptr_t)slot;
if (s->len > SCR_ITABLE_MAX_KEY_BYTES - add) add = SCR_ITABLE_MAX_KEY_BYTES + 1;
else add += s->len;
}
}
if (add > SCR_ITABLE_MAX_KEY_BYTES - n) {
scr_itable_refuse("InternTable array keys can encode at most 2147483647 bytes");
return false;
}
n += add;
}
*out = n;
return true;
}
/* Writes the encoding of `key` into `out`, which holds exactly the size
* scr_itable_encoded_size answered. */
static void scr_itable_encode(const ScrArr *key, int kind, char *out) {
size_t n = 0;
ScrITableElements walk = {key, 0};
for (size_t i = 0; i < key->len; i++) {
uint64_t slot = 0;
bool present = scr_itable_element(key, i, &slot) == SCR_ARR_VALUE;
bool present = scr_itable_element(&walk, i, &slot) == SCR_ARR_VALUE;
if (kind == SCR_ITABLE_KEY_F64S) {
uint64_t bits = SCR_ITABLE_ABSENT_F64;
if (present) {
@@ -483,21 +538,19 @@ static size_t scr_itable_encode(const ScrArr *key, int kind, char *out, size_t c
memcpy(&d, &slot, sizeof d);
bits = scr_itable_f64_bits(d);
}
if (n + 8 <= cap) memcpy(out + n, &bits, 8);
memcpy(out + n, &bits, 8);
n += 8;
} else {
const ScrStr *s = present ? (const ScrStr *)(uintptr_t)slot : NULL;
if (s && s->len >= SCR_ITABLE_ABSENT_STR) scr_trap("scriptc: InternTable key is too long\n");
uint32_t len = s ? (uint32_t)s->len : SCR_ITABLE_ABSENT_STR;
if (n + 4 <= cap) memcpy(out + n, &len, 4);
memcpy(out + n, &len, 4);
n += 4;
if (s) {
if (n + s->len <= cap) memcpy(out + n, s->data, s->len);
memcpy(out + n, s->data, s->len);
n += s->len;
}
}
}
return n;
}
static inline uint64_t scr_itable_bytes_hash(const char *bytes, size_t len) {
@@ -505,14 +558,16 @@ static inline uint64_t scr_itable_bytes_hash(const char *bytes, size_t len) {
}
/* Encodes `key` into `stack` when it fits, else into a heap buffer the
* caller frees. */
* caller frees; NULL (with a pending TypeError) for a refused key. */
static char *scr_itable_encoded(const ScrITable *t, const ScrArr *key, char *stack, size_t cap, size_t *len) {
*len = scr_itable_encode(key, t->key_kind, stack, cap);
if (*len <= cap) return stack;
char *heap = malloc(*len);
if (!heap) scr_itable_oom();
scr_itable_encode(key, t->key_kind, heap, *len);
return heap;
if (!scr_itable_encoded_size(key, t->key_kind, len)) return NULL;
char *bytes = stack;
if (*len > cap) {
bytes = malloc(*len);
if (!bytes) scr_itable_oom();
}
scr_itable_encode(key, t->key_kind, bytes);
return bytes;
}
#define SCR_ITABLE_STACK_KEY 256
@@ -522,6 +577,7 @@ const ScrITableEntry *scr_itable_find_arr(ScrITable *t, const ScrArr *key) {
char stack[SCR_ITABLE_STACK_KEY];
size_t len;
char *bytes = scr_itable_encoded(t, key, stack, sizeof stack, &len);
if (!bytes) return NULL;
uint64_t hash = scr_itable_bytes_hash(bytes, len);
ScrITableSegment *seg = scr_itable_segment(t, hash);
const ScrITableEntry *found = (const ScrITableEntry *)scr_itable_probe_bytes(
@@ -535,6 +591,7 @@ const ScrITableEntry *scr_itable_insert_arr(ScrITable *t, const ScrArr *key, uin
char stack[SCR_ITABLE_STACK_KEY];
size_t len;
char *bytes = scr_itable_encoded(t, key, stack, sizeof stack, &len);
if (!bytes) return NULL;
const ScrITableEntry *entry = scr_itable_insert_bytes(t, scr_itable_bytes_hash(bytes, len), bytes, len, value);
if (bytes != stack) free(bytes);
return entry;
+3 -1
View File
@@ -38,7 +38,9 @@ typedef struct ScrITableEntry {
/* The table called `name`, created on first use. Traps when the name is
* already open with another key kind. Never returns NULL. */
ScrITable *scr_itable_open(const ScrStr *name, int key_kind);
/* The entry of `key`, or NULL. */
/* The entry of `key`, or NULL. The array variants refuse keys of more
* than 65536 elements, or whose encoding exceeds 2^31 - 1 bytes, with a
* pending TypeError (and answer NULL). */
const ScrITableEntry *scr_itable_find_str(ScrITable *t, const ScrStr *key);
const ScrITableEntry *scr_itable_find_f64(ScrITable *t, double key);
const ScrITableEntry *scr_itable_find_arr(ScrITable *t, const ScrArr *key);
+33 -2
View File
@@ -8,12 +8,20 @@
#include <stdlib.h>
#include <string.h>
/* The table unit needs only scr_trap from the rest of the runtime. */
/* The table unit needs only scr_trap and scr_throw_error_msg from the
* rest of the runtime; refusals are recorded here. */
_Noreturn void scr_trap(const char *msg) {
fputs(msg, stderr);
abort();
}
static char refused[128];
void scr_throw_error_msg(int kind, const char *msg, size_t len) {
assert(kind == SCR_ERR_TYPE && len < sizeof refused);
memcpy(refused, msg, len);
refused[len] = 0;
}
enum { THREADS = 8, KEYS = 20000, ROUNDS = 3 };
/* A stand-in for a published value: written by its creator before the
@@ -161,6 +169,19 @@ int main(void) {
big->data = calloc(100, sizeof(uint64_t));
for (unsigned i = 0; i < 100; i++) memcpy(&big->data[i], &(double){i}, sizeof(double));
assert(scr_itable_insert_arr(lists, big, 6)->value == 6 && scr_itable_find_arr(lists, big)->value == 6);
/* Keys of more than 65536 elements are refused before anything is read,
* however long their sparse length. */
ScrArr *longest = make_f64s(NULL, 0);
longest->len = 65536;
assert(scr_itable_insert_arr(lists, longest, 7)->value == 7 && !refused[0]);
ScrArr *huge = make_f64s(NULL, 0);
huge->len = (size_t)1 << 29;
assert(!scr_itable_find_arr(lists, huge) && !strcmp(refused, "InternTable array keys can have at most 65536 elements"));
refused[0] = 0;
longest->len = 65537;
assert(!scr_itable_insert_arr(lists, longest, 8) && refused[0]);
refused[0] = 0;
assert(scr_itable_size(lists) == 6);
ScrITable *paths = scr_itable_open(paths_name, SCR_ITABLE_KEY_STRS);
ScrStr *ab = make_str("ab"), *c = make_str("c"), *a = make_str("a"), *bc = make_str("bc");
ScrArr *p1 = make_strs((ScrStr *[]){ab, c}, 2), *p2 = make_strs((ScrStr *[]){a, bc}, 2);
@@ -168,7 +189,17 @@ int main(void) {
const ScrITableEntry *e1 = scr_itable_insert_arr(paths, p1, 7);
assert(scr_itable_insert_arr(paths, p2, 8) != e1 && scr_itable_find_arr(paths, p3) == e1);
assert(scr_itable_size(paths) == 2);
ScrArr *arrays[] = {one, same, shorter, holed, empty_list, big, p1, p2, p3};
/* String keys whose encoding exceeds 2^31 - 1 bytes are refused. */
ScrStr *wide = calloc(1, sizeof(ScrStr) + 1);
assert(wide);
wide->rc = 1;
wide->len = (size_t)0x7fffffff - 2;
ScrArr *heavy = make_strs((ScrStr *[]){wide, wide}, 2);
assert(!scr_itable_find_arr(paths, heavy) &&
!strcmp(refused, "InternTable array keys can encode at most 2147483647 bytes"));
refused[0] = 0;
free(wide);
ScrArr *arrays[] = {one, same, shorter, holed, empty_list, big, longest, huge, p1, p2, p3, heavy};
for (unsigned i = 0; i < sizeof arrays / sizeof *arrays; i++) {
free(arrays[i]->data);
free(arrays[i]->present);
@@ -0,0 +1,39 @@
import { InternTable } from "@scriptc/experimental";
// InternTable (@scriptc/experimental) array keys can have at most 65536
// elements: longer keys, including sparse arrays with a large length, make
// every method throw a TypeError and store nothing. Node.js prints the same.
const lists = new InternTable<readonly number[], number>("limits.lists");
const names = new InternTable<readonly string[], number>("limits.names");
function attempt(what: string, run: () => unknown): void {
try {
console.log(what, run());
} catch (error) {
console.log(what, (error as Error).name, (error as Error).message);
}
}
const longest = new Array<number>(65536);
longest[0] = 1;
attempt("longest", () => lists.getOrInsert(longest, 1));
attempt("longest again", () => lists.get(longest));
const longer = new Array<number>(65537);
attempt("longer get", () => lists.get(longer));
attempt("longer has", () => lists.has(longer));
attempt("longer insert", () => lists.getOrInsert(longer, 2));
attempt("longer computed", () => lists.getOrInsertComputed(longer, () => 3));
const sparse: number[] = [1, 2];
sparse.length = 2 ** 29;
attempt("sparse get", () => lists.get(sparse));
attempt("sparse insert", () => lists.getOrInsert(sparse, 4));
attempt("sparse computed", () => lists.getOrInsertComputed(sparse, () => 5));
const words: string[] = ["a"];
words.length = 2 ** 30;
attempt("words get", () => names.get(words));
attempt("words insert", () => names.getOrInsert(words, 6));
console.log(lists.size, names.size, lists.get([1]), names.get(["a"]));
+2
View File
@@ -289,6 +289,8 @@ describe.skipIf(!zigOnPath())("wasm32-wasi differential", () => {
// InternTable handles carry 32-bit pointers in f64 slots.
"5720-intern-table-basics.ts",
"5721-intern-table-values.ts",
// Array key sizes are checked in 32-bit size_t arithmetic.
"5726-intern-table-key-limits.ts",
])("the LLVM target runs %s byte-identically to Node", async (fixture) => {
const entry = join(repoRoot, "tests/corpus", fixture);
const outDir = await mkdtemp("/tmp/scriptc-wasi-");