mirror of
https://github.com/cloudflare/cloudflare-os.git
synced 2026-11-23 01:58:45 +08:00
* Build releases in parallel Every worker bundle in a release is independent: each `wrangler deploy --dry-run` reads only its own package and writes only its own outdir. They ran one at a time anyway, and the frontend build sat in front of all of them. Measured on a 10-core machine: 62.4s -> 12.3s. The frontend now overlaps with the bundles too. Only a worker that serves static assets reads workshop-frontend/dist, so it alone awaits the frontend build -- read off `config.assets` rather than naming the router, so a second asset-serving worker would be sequenced as well. Nothing about the output depends on the interleaving: results come back in `findDeployablePackages` order, and the blob writes and the manifest are built from them in that order. Verified byte-identical against a sequential baseline (modules, assets and manifest, modulo the `createdAt` timestamp), at the default concurrency and at `--concurrency 1`. `mapConcurrent` is a shared, tested helper rather than a `Promise.all` at the call site for its failure behaviour: `Promise.all` settles on the first rejection and leaves the rest bundling under a process that is already exiting, so CI would report whichever package lost the race. Here every task is awaited and all the failures are named together. `run` captures output instead of inheriting the terminal -- a dozen wrangler logs interleaved line by line would be unreadable -- and prints one labelled block per command with its duration, in completion order. A failed command's diagnostics are in its block, so the summary carries only labels and exit codes rather than repeating them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Cancel in-flight builds on the first failure, and flush diagnostics before exiting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
65 lines
3.2 KiB
TypeScript
65 lines
3.2 KiB
TypeScript
// Runs an async operation over a list with a bounded number of them in flight.
|
|
//
|
|
// Written for the release build, where each package's `wrangler deploy --dry-run` is independent
|
|
// and saturates about one core. Two properties matter there, and they are why this is a shared,
|
|
// tested helper rather than a `Promise.all` at the call site:
|
|
//
|
|
// - Results keep the input's order, so a manifest built from them is byte-stable no matter which
|
|
// bundle happened to finish first.
|
|
// - A rejection does not abandon the tasks still running. `Promise.all` settles on the first
|
|
// failure and leaves the rest writing to disk under a process that is already exiting; here
|
|
// every task is awaited, and all the failures are reported together -- in CI that names every
|
|
// broken package rather than whichever one lost the race.
|
|
//
|
|
// The optional signal is the fail-fast complement: when the caller learns elsewhere that the whole
|
|
// run is doomed (the frontend build failed), aborting stops new tasks from being claimed while the
|
|
// wait-for-all guarantee on the ones already running still holds.
|
|
|
|
/**
|
|
* Applies `task` to every item, with at most `limit` running at a time, and returns the results in
|
|
* the input's order.
|
|
*
|
|
* Every task runs even if an earlier one rejects. If any did, this rejects once they have all
|
|
* settled: with that error if there was exactly one, or an `AggregateError` over them otherwise.
|
|
*
|
|
* An aborted `signal` stops further items from starting; tasks already running are still awaited.
|
|
* If the signal aborted at any point and no task failed, this rejects with `signal.reason`; task
|
|
* failures take precedence over the abort reason.
|
|
*/
|
|
export async function mapConcurrent<T, R>(
|
|
items: readonly T[],
|
|
limit: number,
|
|
task: (item: T, index: number) => Promise<R>,
|
|
signal?: AbortSignal,
|
|
): Promise<R[]> {
|
|
if (!Number.isInteger(limit) || limit < 1) {
|
|
throw new RangeError(`concurrency limit must be a positive integer, got ${limit}`);
|
|
}
|
|
const results: R[] = Array.from({ length: items.length });
|
|
const failures: unknown[] = [];
|
|
// Each runner claims the next index and works until the list is exhausted. The claim is a bare
|
|
// `next++` because JS runs it to completion before any other runner resumes.
|
|
let next = 0;
|
|
const runners = Array.from({ length: Math.min(limit, items.length) }, async () => {
|
|
for (let i = next++; i < items.length; i = next++) {
|
|
if (signal?.aborted) break;
|
|
try {
|
|
results[i] = await task(items[i], i);
|
|
} catch (error) {
|
|
failures.push(error);
|
|
}
|
|
}
|
|
});
|
|
await Promise.all(runners);
|
|
if (failures.length === 1) throw failures[0];
|
|
if (failures.length > 1) {
|
|
throw new AggregateError(failures, `${failures.length} of ${items.length} tasks failed`);
|
|
}
|
|
// Aborted is aborted, whether or not it cost us an item. Keying this off "we skipped something"
|
|
// instead reports success for the case where the signal lands once every item is already claimed
|
|
// and each task then finishes on its own merits -- no holes in `results`, nothing rejected, and a
|
|
// caller that asked to stop handed a full set of results as if it had not.
|
|
if (signal?.aborted) throw signal.reason;
|
|
return results;
|
|
}
|