mirror of
https://github.com/cloudflare/cloudflare-os.git
synced 2026-11-23 01:58:45 +08:00
* Await the share-link rejection assertions that never ran Four tests built an expect(...).rejects chain without awaiting it, so the assertion settled after the test had already passed. Vitest 5 fails such floating assertions; await them so they actually check something. * Move the test toolchain to Vitest 5 and @cloudflare/vitest-plugin @vitest/browser-playwright 5 needs Vitest 5, and @cloudflare/vitest-pool-workers never supported it: the package was renamed @cloudflare/vitest-plugin, which added Vitest 5 in 1.4.0. Rename it everywhere and move vitest to ^5.0.5. vite-plus goes to ^1.2.0, which bundles Vitest 5.0.4 rather than 1.0.0's 5.0.1. 1.3.0 bundles 5.0.5 and would remove the last duplicate copy, but is still inside the release-age window. The plugin keeps running on the current Wrangler/Miniflare stack: the cloudflare-toolchain bumps (#785) make google-hooks-upgrade flaky (ECONNRESET, 3-5 failures in 8 runs against none on main) and are left for their own change. vitest-evals 0.17 still declares vitest <5 but works unchanged on it (harness, judges, reporter and JSON output checked), so a peerDependencyRules entry allows it until a release widens the range. Supersedes #791 and #786. * Run vp's vitest tasks without Vitest 5's filesystem module cache Vitest 5 turns fsModuleCache on by default, and every package's vitest shares one cache under the workspace root's node_modules. vp runs those tasks in parallel, so when one run invalidates the cache it deletes the directory while another is writing into it, and that run fails to start with ENOTEMPTY. vp already caches the whole task, so the builder appends --no-fsModuleCache to each vitest command, as it already pins the reporter. * Turn off Vitest 5's filesystem module cache for the backend's Workers suite vp's tasks already run without it, but a direct vitest run in the package still uses it, and under the Workers pool every run after the first fails: cached .txt modules come back with root-relative ids the textModules plugin cannot read, and the pool's WebSocket rejects the larger cached modules (1009, Message is too large).
320 lines
14 KiB
JavaScript
Executable File
320 lines
14 KiB
JavaScript
Executable File
#!/usr/bin/env node
|
|
// Run a command under two watchdogs -- an idle timer on its output and a total wall-clock cap --
|
|
// and kill its whole process tree when either fires.
|
|
//
|
|
// node ../../scripts/with-timeout.ts --idle <secs> --max <secs> -- <command> [args...]
|
|
//
|
|
// It exists because nothing else in the test stack bounds a wedged run. Vite+ has no task timeout
|
|
// (`vp run --help` offers only concurrency flags, and `timeout` appears nowhere in its docs), and
|
|
// vitest's `testTimeout`/`hookTimeout` are enforced *inside* the test worker, so they die with it;
|
|
// the one process-level watchdog, `teardownTimeout`, is armed after the run resolves and therefore
|
|
// never arms when the run is what hangs. Underneath that,
|
|
// `@cloudflare/vitest-plugin` imports Miniflare without passing `onWorkerdCrashRestart`, so a
|
|
// workerd that dies mid-run leaves the pool awaiting a reply that will never arrive. The result is
|
|
// an unbounded local hang, and in CI an anonymous job-level kill that names no culprit.
|
|
//
|
|
// Silence is the primary detector rather than wall-clock: a healthy `vitest run` prints a line per
|
|
// completed test file, and a wedged one goes quiet. The total cap is the backstop for a command
|
|
// that stays chatty while looping forever.
|
|
//
|
|
// It runs as two processes. When a task fails, vp's fast-fail SIGKILLs the direct child of every
|
|
// other running task; were that child the watchdog, its tree would run on unbounded, and on Linux
|
|
// vp waits for it with no timeout. So the process vp spawns is a front that re-runs this file with
|
|
// `--lifeline` as the watchdog (the back, `supervise`), holding an IPC channel to it, and the back
|
|
// kills the tree when that channel drops, however the front died. The back stays the command's
|
|
// output reader, since an orphaned vitest dies on its first write to a dead pipe and its forks
|
|
// escape the tree walk. The front forwards no signal: Ctrl-C already reaches the back and the tree,
|
|
// and a second copy would cut the grace period short.
|
|
//
|
|
// Exit codes: 124 when a threshold fired (GNU `timeout`'s, and distinct from any vitest code), the
|
|
// child's own code otherwise, or `128 + signal` when the child or this wrapper was signalled.
|
|
//
|
|
// `TESTS_WITH_TIMEOUT_DISABLE=<anything>` switches both watchdogs off, leaving the stdio piping
|
|
// and signal relay: for a CI whose job-level timeout already bounds a wedge, or a deliberately
|
|
// unsupervised local run. A wedge is then killed anonymously by whatever is outside, with no
|
|
// surviving-tree listing to name the culprit. Under vp the variable reaches a cached task only
|
|
// through the task's `env` declaration (see vitest-task-vite-config.ts), which also fingerprints it.
|
|
// `TESTS_WITH_TIMEOUT_IDLE_SECONDS` and `TESTS_WITH_TIMEOUT_MAX_SECONDS` override their respective
|
|
// CLI thresholds for slow CI/container runs. Both require positive, finite seconds and are likewise
|
|
// forwarded and fingerprinted; neither disables the watchdog.
|
|
|
|
import { execFile, spawn } from "node:child_process";
|
|
import { constants } from "node:os";
|
|
import { fileURLToPath } from "node:url";
|
|
import { resolveBinEntry } from "./bin-entry.ts";
|
|
import { collectTree, killProcessTreeEscalating } from "./kill-process-tree.ts";
|
|
|
|
/** Exit status for a threshold firing, matching GNU `timeout`. */
|
|
const TIMED_OUT_EXIT_CODE = 124;
|
|
|
|
/** How long a signalled tree gets to exit on its own before the SIGKILL. */
|
|
const KILL_GRACE_MS = 5_000;
|
|
|
|
/** Marks the back's invocation by the front (see the header). Internal, not part of the CLI. */
|
|
const LIFELINE_FLAG = "--lifeline";
|
|
|
|
const USAGE = "usage: with-timeout.ts --idle <secs> --max <secs> -- <command> [args...]";
|
|
|
|
type Options = {
|
|
idleMs: number;
|
|
maxMs: number;
|
|
argv: string[];
|
|
};
|
|
|
|
// Both thresholds are required rather than defaulted: the numbers are policy, and a caller that
|
|
// forgot one should not silently get a different bound than it asked for. Fractional seconds are
|
|
// accepted so the tests can drive this with sub-second thresholds.
|
|
function parseArgs(args: string[]): Options {
|
|
let idleMs: number | undefined;
|
|
let maxMs: number | undefined;
|
|
let index = 0;
|
|
for (; index < args.length; index++) {
|
|
const flag = args[index];
|
|
if (flag === "--") {
|
|
index++;
|
|
break;
|
|
}
|
|
const value = args[++index];
|
|
if (flag === "--idle") idleMs = parseSeconds(flag, value);
|
|
else if (flag === "--max") maxMs = parseSeconds(flag, value);
|
|
else fail(`unknown option ${flag}\n${USAGE}`);
|
|
}
|
|
const argv = args.slice(index);
|
|
if (idleMs === undefined || maxMs === undefined) fail(`--idle and --max are both required\n${USAGE}`);
|
|
if (argv.length === 0) fail(`no command given\n${USAGE}`);
|
|
return { idleMs, maxMs, argv };
|
|
}
|
|
|
|
function parseSeconds(flag: string, value: string | undefined): number {
|
|
const seconds = Number(value);
|
|
if (!Number.isFinite(seconds) || seconds <= 0) fail(`${flag} needs a positive number of seconds`);
|
|
return seconds * 1000;
|
|
}
|
|
|
|
function fail(message: string): never {
|
|
process.stderr.write(`with-timeout: ${message}\n`);
|
|
process.exit(2);
|
|
}
|
|
|
|
/**
|
|
* The command to spawn, resolved the way `run-dev-server.ts` resolves the binaries it launches:
|
|
* `node` goes to this process's own executable, a workspace bin is reached as `node <entry>` so no
|
|
* shell is needed for the `.bin` shim, and anything else is spawned as written for PATH to resolve.
|
|
*/
|
|
function resolveCommand(argv: string[]): [string, string[]] {
|
|
const [bin, ...rest] = argv;
|
|
if (bin === "node") return [process.execPath, rest];
|
|
const entry = resolveBinEntry(process.cwd(), bin);
|
|
return entry ? [process.execPath, [entry, ...rest]] : [bin, rest];
|
|
}
|
|
|
|
/**
|
|
* The surviving process tree under `pid`, as `ps` prints it. Empty on Windows, where the tree is
|
|
* killed by `taskkill /T` rather than by walking it.
|
|
*
|
|
* The distinction this buys, without having to reproduce the hang: "vitest alive, zero workerd
|
|
* children" -- the documented crash wedge -- versus "workerd alive but not answering".
|
|
*/
|
|
async function describeTree(pid: number): Promise<string> {
|
|
if (process.platform === "win32") return "";
|
|
// Walked before anything is signalled: killing the parent reparents its descendants, after which
|
|
// the child listers find nothing (see the note in kill-process-tree.ts). The escalating kill
|
|
// walks again afterwards, and that second walk -- not this one -- is what drives its SIGKILL.
|
|
const pids = await collectTree(pid);
|
|
const listing = await new Promise<string>(resolve => {
|
|
execFile("ps", ["-o", "pid=,command=", "-p", pids.join(",")], (_error, stdout) => {
|
|
resolve(stdout.trimEnd());
|
|
});
|
|
});
|
|
const detail = listing || ` (ps reported nothing for ${pids.join(", ")})`;
|
|
return ` surviving processes (${pids.length}):\n${detail}\n`;
|
|
}
|
|
|
|
/**
|
|
* Resolves once everything written to stdout and stderr has drained. `process.exit()` truncates a
|
|
* pending write when the stream is a pipe, which is exactly what it is under `vp run`.
|
|
*/
|
|
async function flushOutput(): Promise<void> {
|
|
await Promise.all([process.stdout, process.stderr].map(
|
|
stream => new Promise<void>(resolve => { stream.write("", () => resolve()); })));
|
|
}
|
|
|
|
function exitCodeForSignal(signal: NodeJS.Signals): number {
|
|
return 128 + (constants.signals[signal] ?? 0);
|
|
}
|
|
|
|
/** The front (see the header): runs the back over the lifeline and exits with its status. */
|
|
function holdLifeline(args: string[]): void {
|
|
const back = spawn(process.execPath, [fileURLToPath(import.meta.url), LIFELINE_FLAG, ...args], {
|
|
stdio: ["inherit", "inherit", "inherit", "ipc"],
|
|
});
|
|
|
|
// A signalled front reports `128 + signal`, whatever the back's teardown ends on.
|
|
let signalled: NodeJS.Signals | undefined;
|
|
for (const signal of ["SIGINT", "SIGTERM"] as const) {
|
|
process.on(signal, () => {
|
|
signalled ??= signal;
|
|
if (back.connected) back.disconnect();
|
|
});
|
|
}
|
|
|
|
back.on("error", error => {
|
|
process.stderr.write(`with-timeout: could not start the watchdog: ${error.message}\n`);
|
|
process.exit(127);
|
|
});
|
|
|
|
// `exit` rather than `close`: the back's stdio is inherited, so there is no pipe here to drain.
|
|
back.on("exit", (code, signal) => {
|
|
if (signalled) process.exit(exitCodeForSignal(signalled));
|
|
process.exit(signal ? exitCodeForSignal(signal) : code ?? 1);
|
|
});
|
|
}
|
|
|
|
/** The back (see the header): the watchdog itself, run by the front with `--lifeline`. */
|
|
function supervise(args: string[]): void {
|
|
const defaults = parseArgs(args);
|
|
const idleOverride = process.env.TESTS_WITH_TIMEOUT_IDLE_SECONDS;
|
|
const maxOverride = process.env.TESTS_WITH_TIMEOUT_MAX_SECONDS;
|
|
const idleMs = idleOverride === undefined ? defaults.idleMs :
|
|
parseSeconds("TESTS_WITH_TIMEOUT_IDLE_SECONDS", idleOverride);
|
|
const maxMs = maxOverride === undefined ? defaults.maxMs :
|
|
parseSeconds("TESTS_WITH_TIMEOUT_MAX_SECONDS", maxOverride);
|
|
const { argv } = defaults;
|
|
// After `parseArgs`, so a malformed invocation fails with the usage error even when disabled.
|
|
const watchdogsDisabled = Boolean(process.env.TESTS_WITH_TIMEOUT_DISABLE);
|
|
const [command, commandArgs] = resolveCommand(argv);
|
|
const startedAt = Date.now();
|
|
|
|
// A reader that goes away must not take the watchdog down with it: an unhandled EPIPE on our own
|
|
// stdio is an uncaught exception, and this process would exit 1 with the wedged tree still
|
|
// running. There is nowhere to report a failure to write to stderr anyway.
|
|
for (const stream of [process.stdout, process.stderr]) stream.on("error", () => {});
|
|
|
|
// The command inherits neither the channel (close-on-exec) nor `NODE_CHANNEL_FD` (deleted).
|
|
const child = spawn(command, commandArgs, { stdio: ["inherit", "pipe", "pipe"] });
|
|
|
|
// Set as soon as a threshold, a signal or the lifeline takes over, so the child's own `close` --
|
|
// which follows the kill -- does not report an exit status over the one already chosen.
|
|
let terminating = false;
|
|
let idleTimer: NodeJS.Timeout | undefined;
|
|
let maxTimer: NodeJS.Timeout | undefined;
|
|
|
|
function clearTimers(): void {
|
|
clearTimeout(idleTimer);
|
|
clearTimeout(maxTimer);
|
|
}
|
|
|
|
// Also the reset on every output chunk (see `forward`), so the disabled check lives here rather
|
|
// than only at the initial arm.
|
|
function armIdleTimer(): void {
|
|
clearTimeout(idleTimer);
|
|
if (terminating || watchdogsDisabled) return;
|
|
idleTimer = setTimeout(() => { void onThreshold("no output", idleMs); }, idleMs);
|
|
}
|
|
|
|
// Straight through to this process's own streams, resetting the idle timer on every chunk. vp
|
|
// captures a task's output rather than handing over the TTY -- it has to, to render its per-task
|
|
// tree and replay a cache hit -- so interposing this pipe costs nothing the run was not paying.
|
|
//
|
|
// `pipe` rather than a write per chunk so a slow reader applies backpressure to the child instead
|
|
// of accumulating in here; the `data` listener beside it is only the idle reset. `end: false`
|
|
// because this process's own stdio must outlive the child -- the diagnostic is written after it.
|
|
function forward(source: NodeJS.ReadableStream, sink: NodeJS.WritableStream): void {
|
|
source.on("data", () => armIdleTimer());
|
|
source.pipe(sink, { end: false });
|
|
}
|
|
|
|
async function onThreshold(reason: string, thresholdMs: number): Promise<void> {
|
|
if (terminating || child.pid === undefined) return;
|
|
terminating = true;
|
|
clearTimers();
|
|
const elapsed = ((Date.now() - startedAt) / 1000).toFixed(1);
|
|
process.stderr.write(
|
|
`\nwith-timeout: ${reason} for ${thresholdMs / 1000}s -- killing the process tree\n` +
|
|
` command: ${argv.join(" ")}\n` +
|
|
` cwd: ${process.cwd()}\n` +
|
|
` elapsed: ${elapsed}s\n` +
|
|
await describeTree(child.pid));
|
|
await killProcessTreeEscalating(child.pid, { graceMs: KILL_GRACE_MS });
|
|
await flushOutput();
|
|
process.exit(TIMED_OUT_EXIT_CODE);
|
|
}
|
|
|
|
// A second signal gives up the rest of the grace period and escalates now. Bounding the Ctrl-C
|
|
// path matters as much as bounding the run: the reported hang ended with the contributor
|
|
// interrupting it, and a tree that ignores the interrupt is what left vp escalating to SIGKILL on
|
|
// its own.
|
|
const force = new AbortController();
|
|
let signalled = false;
|
|
let lifelineLost = false;
|
|
|
|
async function onSignal(signal: NodeJS.Signals): Promise<void> {
|
|
if (signalled) {
|
|
force.abort();
|
|
return;
|
|
}
|
|
signalled = true;
|
|
// On Ctrl-C the front's disconnect can land first; that teardown is this signal's.
|
|
if (lifelineLost) return;
|
|
terminating = true;
|
|
clearTimers();
|
|
if (child.pid !== undefined) {
|
|
await killProcessTreeEscalating(
|
|
child.pid, { graceMs: KILL_GRACE_MS, forceSignal: force.signal });
|
|
}
|
|
await flushOutput();
|
|
process.exit(exitCodeForSignal(signal));
|
|
}
|
|
|
|
// The front is gone or signalled, and reports any status itself, so this one is only nonzero.
|
|
async function onLifelineLost(): Promise<void> {
|
|
lifelineLost = true;
|
|
if (terminating || child.pid === undefined) return;
|
|
terminating = true;
|
|
clearTimers();
|
|
await killProcessTreeEscalating(
|
|
child.pid, { graceMs: KILL_GRACE_MS, forceSignal: force.signal });
|
|
await flushOutput();
|
|
process.exit(1);
|
|
}
|
|
|
|
for (const signal of ["SIGINT", "SIGTERM"] as const) {
|
|
process.on(signal, () => { void onSignal(signal); });
|
|
}
|
|
|
|
// Unref'd so only the child holds this process open.
|
|
process.channel?.unref();
|
|
process.on("disconnect", () => { void onLifelineLost(); });
|
|
|
|
child.on("error", error => {
|
|
process.stderr.write(`with-timeout: could not start ${argv.join(" ")}: ${error.message}\n`);
|
|
process.exit(127);
|
|
});
|
|
|
|
// `close` rather than `exit`: it fires once the piped stdio has been drained too, so nothing the
|
|
// child printed on its way out is dropped.
|
|
child.on("close", (code, signal) => {
|
|
if (terminating) return;
|
|
clearTimers();
|
|
void flushOutput().then(() => {
|
|
process.exit(signal ? exitCodeForSignal(signal) : code ?? 1);
|
|
});
|
|
});
|
|
|
|
if (child.stdout) forward(child.stdout, process.stdout);
|
|
if (child.stderr) forward(child.stderr, process.stderr);
|
|
if (!watchdogsDisabled) {
|
|
armIdleTimer();
|
|
maxTimer = setTimeout(() => { void onThreshold("still running", maxMs); }, maxMs);
|
|
}
|
|
}
|
|
|
|
const args = process.argv.slice(2);
|
|
if (args[0] !== LIFELINE_FLAG) {
|
|
holdLifeline(args);
|
|
} else if (!process.channel) {
|
|
fail(`${LIFELINE_FLAG} is internal: it is how the front runs the watchdog over an IPC channel`);
|
|
} else {
|
|
supervise(args.slice(1));
|
|
}
|