diff --git a/docs/content/docs/how-it-works.mdx b/docs/content/docs/how-it-works.mdx
index 3747679c4..0475e311e 100644
--- a/docs/content/docs/how-it-works.mdx
+++ b/docs/content/docs/how-it-works.mdx
@@ -34,7 +34,7 @@ Runtime pack and linker -> executable
## Build parallelism
-Most of a build runs on one CPU core. The LLVM helper uses more cores only when it compiles an executable in partitions.
+Most of a build runs on one CPU core. The LLVM helper uses more cores when it compiles an executable in partitions, and when it generates code for an optimized executable of one partition.
### Frontend
@@ -62,7 +62,7 @@ The count depends only on the size of the LLVM text, so a program has the same p
These builds are not divided into partitions and compile on one thread: `--emit=asm`, `--emit=obj`, and `--print=native-link-info` output, library builds (`--lib`; `dev` library builds are compiled in pieces, one after another), and `--sanitize` builds, which compile the LLVM IR with the C compiler.
-The helper compiles partitions on a pool with one thread per hardware thread of the host, so an executable with four partitions uses at most four cores in the backend. A program of one partition is optimized and compiled on one thread.
+The helper compiles partitions on a pool with one thread per hardware thread of the host, so an executable with four partitions uses at most four cores in the backend.
How partitions are compiled depends on `--optimization`:
@@ -71,6 +71,23 @@ How partitions are compiled depends on `--optimization`:
Reading the LLVM text and dividing it into partitions, and the import step, run on one thread.
+### Code generation of one partition
+
+A `release` or `speed` executable of one partition is optimized as one module on one thread. When its LLVM text is larger than 128 KiB, the helper then divides the optimized module for code generation and generates machine code for the parts in parallel, on the same pool. Each part holds a run of consecutive functions with about the same amount of code, so the executable keeps the functions in the same order. The module is optimized before it is divided, so each function's machine code is the same as from one thread, except that a jump to a function that another part also uses takes the longer form. When one function holds most of the code, the first part holds the whole module and the others are empty.
+
+
+
+
LLVM text
Code generation parts
+
+
+
Up to 128 KiB
1
+
Over 128 KiB, up to 256 KiB
4
+
Over 256 KiB, up to 1 MiB
8
+
+
+
+Like partitions, the count depends only on the size of the LLVM text. A `dev` executable of one partition is compiled on one thread.
+
### Rebuilds
With two or more partitions, the helper keeps each partition's summary and object in the build cache (see `SCRIPTC_CACHE_DIR` in the [CLI reference](/docs/cli#environment-variables)). A rebuild after an edit compiles again only the partitions whose functions changed and, in `release` and `speed` builds, the partitions that import a changed function; the others come from the cache. Dividing the program, the import step, and linking run on every build. A cold build, with nothing cached, compiles every partition.
diff --git a/native/llvm-codegen/README.md b/native/llvm-codegen/README.md
index 0af8a11d2..7305db21d 100644
--- a/native/llvm-codegen/README.md
+++ b/native/llvm-codegen/README.md
@@ -35,6 +35,8 @@ requested output.
Repeating `--output` divides the program into that many partitions, which are compiled concurrently and linked together. The helper splits the emitter's LLVM assembly text directly, so partitions also parse concurrently; input outside the emitter's narrow grammar is parsed whole and split with LLVM instead. Internal definitions that no other definition reaches are left out of every partition, and the others become hidden so partitions can reference each other. At opt-level 0 each partition is compiled independently. Optimized partitions follow the ThinLTO model: each is simplified and summarized on its own thread, a thin link decides which small functions every partition imports for inlining and which hidden definitions become internal again, and each partition is then optimized and compiled on its own thread. Partition placement depends only on the module and the partition count, so output never depends on the host.
+With `--split codegen`, several outputs divide only code generation. The helper optimizes the module whole, as it does for a single output, then divides its functions in module order into runs of about the same instruction count, one per output, and generates code for every output on its own thread. Linking the outputs in order lays the functions out as one output would. A local definition that another output uses becomes hidden so the objects can reference each other; the others stay local. Every output after the first is generated from a copy of the optimized module read back from bitcode with its use-list order, so each function's machine code matches a single output except for jumps to definitions that another output also uses. When one run would hold more than three quarters of the module's instructions, the first output holds the whole module and the other outputs are empty objects. Placement depends only on the optimized module and the output count. `--split codegen` requires an optimized build and no `--cache-dir`; scriptc passes it for optimized executables with more than 128 KiB of LLVM text that stay one partition.
+
WebAssembly code generation differs from LLVM's default pipeline in two ways that do not change its output. The pass that gives multiple-entry loops a single entry runs only on functions whose control flow is irreducible; its reachability analysis grows with the square of a function's block count even when it has nothing to change. Object emission also leaves out the module pass that collects external symbols for assembly declarations, so each function's machine code is freed once it is printed instead of when the whole module is compiled.
`--cache-dir ` names a private directory the caller owns for one helper build. Simplified partitions are stored under the hash of their exact source and objects under the hash of the exact module after importing, each with a trailing digest and published by rename, so a rebuild recompiles only the partitions an edit can affect. Partitions drop the names of their struct types before the thin link, which would otherwise number same-named types from different partitions in an order that any edit can shift, changing the module after importing in partitions the edit did not touch.
diff --git a/native/llvm-codegen/src/emit.cpp b/native/llvm-codegen/src/emit.cpp
index a263ee601..1ab0f6ecd 100644
--- a/native/llvm-codegen/src/emit.cpp
+++ b/native/llvm-codegen/src/emit.cpp
@@ -6,8 +6,11 @@
#include "runtime-import.h"
#include "target.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallString.h"
#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/StringMap.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/StringSet.h"
#include "llvm/Analysis/RuntimeLibcallInfo.h"
@@ -48,6 +51,7 @@
#include "llvm/TargetParser/Triple.h"
#include "llvm/Transforms/Utils/SplitModule.h"
+#include
#include
#include
#include
@@ -100,6 +104,8 @@ std::optional parseEmitOptions(int Argc, char **Argv) {
Options.ProfileGenerate = Value.str();
else if (Arg == "--profile-use")
Options.ProfileUse = Value.str();
+ else if (Arg == "--split" && Value == "codegen")
+ Options.SplitCodegen = true;
else
return std::nullopt;
}
@@ -1043,6 +1049,249 @@ int emitPartitions(const EmitOptions &Options) {
return 0;
}
+// An optimized module prepared for split code generation.
+struct CodegenSplit {
+ // The module as bitcode, with the order of every use list.
+ SmallVector Bitcode;
+ // The output that holds each definition, by name.
+ StringMap PartOf;
+};
+
+// Divides an optimized module's definitions among Count outputs for code
+// generation. Functions are divided in module order into runs of about the
+// same instruction count, so linking the outputs in order lays the functions
+// out as one module would; a function that would take its run past an even
+// share starts the next run. A local definition that another output uses
+// becomes hidden so the outputs can reference each other; the others stay
+// local. Variables and aliases go with output 0 unless a comdat, an aliasee
+// or an ifunc resolver places them. When one run would hold more than three
+// quarters of the instructions, dividing cannot shorten code generation: the
+// result is empty and M is unchanged. Placement depends only on the module.
+std::optional splitForCodegen(Module &M, size_t Count) {
+ uint64_t Total = 0;
+ for (const Function &F : M)
+ if (!F.isDeclaration())
+ Total += F.getInstructionCount() + 1;
+ DenseMap Part;
+ DenseMap ComdatPart;
+ std::vector Load(Count, 0);
+ uint64_t Share = (Total + Count - 1) / Count;
+ size_t Current = 0;
+ uint64_t InRun = 0;
+ for (Function &F : M) {
+ if (F.isDeclaration())
+ continue;
+ uint64_t Size = F.getInstructionCount() + 1;
+ if (InRun > 0 && InRun + Size > Share && Current + 1 < Count) {
+ ++Current;
+ InRun = 0;
+ }
+ size_t Target = Current;
+ if (const Comdat *C = F.getComdat()) {
+ auto [It, Inserted] = ComdatPart.try_emplace(C, Target);
+ Target = It->second;
+ }
+ Part[&F] = Target;
+ Load[Target] += Size;
+ InRun += Size;
+ }
+ if (*std::max_element(Load.begin(), Load.end()) * 4 > Total * 3)
+ return std::nullopt;
+ auto PartOf = [&](const GlobalValue *GV) -> size_t {
+ if (const GlobalObject *Root = GV->getAliaseeObject()) {
+ if (const auto *IFunc = dyn_cast(Root))
+ Root = IFunc->getResolverFunction();
+ if (Root != nullptr && Root != GV)
+ GV = Root;
+ }
+ if (auto It = Part.find(GV); It != Part.end())
+ return It->second;
+ if (const auto *GO = dyn_cast(GV))
+ if (const Comdat *C = GO->getComdat())
+ if (auto It = ComdatPart.find(C); It != ComdatPart.end())
+ return It->second;
+ return 0;
+ };
+ // Whether a user of GV, through any constants, lies outside output Home.
+ auto UsedElsewhere = [&](const GlobalValue &GV, size_t Home) {
+ SmallVector Work(GV.users());
+ SmallPtrSet Seen;
+ while (!Work.empty()) {
+ const User *U = Work.pop_back_val();
+ if (!Seen.insert(U).second)
+ continue;
+ if (const auto *I = dyn_cast(U)) {
+ if (PartOf(I->getFunction()) != Home)
+ return true;
+ } else if (const auto *G = dyn_cast(U)) {
+ if (PartOf(G) != Home)
+ return true;
+ } else {
+ append_range(Work, U->users());
+ }
+ }
+ return false;
+ };
+ for (GlobalValue &GV : M.global_values()) {
+ if (GV.hasLocalLinkage() && UsedElsewhere(GV, PartOf(&GV))) {
+ GV.setLinkage(GlobalValue::ExternalLinkage);
+ GV.setVisibility(GlobalValue::HiddenVisibility);
+ }
+ if (!GV.hasName())
+ GV.setName("scr.split");
+ }
+ CodegenSplit Split;
+ for (const GlobalValue &GV : M.global_values())
+ if (!GV.isDeclaration())
+ Split.PartOf[GV.getName()] = PartOf(&GV);
+ raw_svector_ostream Stream(Split.Bitcode);
+ WriteBitcodeToFile(M, Stream, /*ShouldPreserveUseListOrder=*/true);
+ return Split;
+}
+
+// Reduces P, a copy of the split module, to the definitions of output I:
+// the other definitions become declarations, and declarations that nothing
+// uses are removed. A declaration nothing uses would still emit an untyped
+// undefined symbol for its visibility, which some linkers reject against a
+// thread-local definition. Removing definitions keeps the order of the
+// remaining uses, so each function of the output reaches code generation
+// as it would in the whole module.
+void keepCodegenPart(Module &P, const StringMap &PartOf, size_t I) {
+ auto Elsewhere = [&](const GlobalValue &GV) {
+ auto It = PartOf.find(GV.getName());
+ return It != PartOf.end() && It->second != I;
+ };
+ // Aliases and ifuncs cannot be declarations, so a declaration replaces
+ // them.
+ SmallVector Indirect;
+ for (GlobalAlias &GA : P.aliases())
+ if (Elsewhere(GA))
+ Indirect.push_back(&GA);
+ for (GlobalIFunc &GI : P.ifuncs())
+ if (Elsewhere(GI))
+ Indirect.push_back(&GI);
+ for (GlobalValue *GV : Indirect) {
+ GlobalValue *Declaration;
+ if (auto *Type = dyn_cast(GV->getValueType()))
+ Declaration = Function::Create(Type, GlobalValue::ExternalLinkage,
+ GV->getAddressSpace(), "", &P);
+ else
+ Declaration = new GlobalVariable(
+ P, GV->getValueType(), /*isConstant=*/false,
+ GlobalValue::ExternalLinkage, nullptr, "", nullptr,
+ GV->getThreadLocalMode(), GV->getAddressSpace());
+ Declaration->setVisibility(GV->getVisibility());
+ Declaration->setDSOLocal(GV->isDSOLocal());
+ Declaration->takeName(GV);
+ GV->replaceAllUsesWith(Declaration);
+ GV->eraseFromParent();
+ }
+ for (Function &F : P)
+ if (!F.isDeclaration() && Elsewhere(F)) {
+ F.deleteBody();
+ F.setComdat(nullptr);
+ }
+ for (GlobalVariable &GV : P.globals())
+ if (!GV.isDeclaration() && Elsewhere(GV)) {
+ GV.setInitializer(nullptr);
+ GV.setComdat(nullptr);
+ GV.setLinkage(GlobalValue::ExternalLinkage);
+ }
+ if (I != 0)
+ P.setModuleInlineAsm("");
+ // Constant expressions that only the removed bodies used remain as users.
+ for (GlobalVariable &GV : make_early_inc_range(P.globals()))
+ if (GV.isDeclaration()) {
+ GV.removeDeadConstantUsers();
+ if (GV.use_empty())
+ GV.eraseFromParent();
+ }
+ for (Function &F : make_early_inc_range(P))
+ if (F.isDeclaration()) {
+ F.removeDeadConstantUsers();
+ if (F.use_empty())
+ F.eraseFromParent();
+ }
+}
+
+// Optimizes M whole, as a single output would, then divides its definitions
+// among the outputs and generates code for each on its own thread. The first
+// output is generated from M with Machine; the others from copies of M read
+// back from bitcode, which keeps the order of every use list, so each
+// function's code matches a single-output build except where it refers to a
+// definition in another output. Placement depends only
+// on the optimized module and the output count. When splitForCodegen keeps
+// the module whole, the other outputs are empty objects.
+std::optional
+emitSplitCodegen(const EmitOptions &Options, Module &M, TargetMachine &Machine,
+ OptimizationLevel Level, ProfileMatch &Match) {
+ bool Annotating = !Options.ProfileUse.empty();
+ if (Annotating)
+ M.getContext().setDiagnosticHandler(
+ std::make_unique(Match));
+ {
+ Pipeline P(Machine, Level, profileOptions(Options));
+ if (Annotating)
+ P.countAnnotated(Match.Annotated);
+ P.PB.buildPerModuleDefaultPipeline(Level).run(M, P.MAM);
+ }
+ if (std::optional Failure =
+ verify(M, "post_optimization_verification_failed"))
+ return Failure;
+ size_t Count = Options.Outputs.size();
+ std::optional Split = splitForCodegen(M, Count);
+ std::string Identifier = M.getModuleIdentifier();
+ Triple TargetTriple = M.getTargetTriple();
+ DataLayout Layout = M.getDataLayout();
+ std::vector>> Temporaries(Count);
+ SharedFailure Failures;
+ forEachPartition(Count, [&](size_t I) {
+ if (I == 0) {
+ if (Split)
+ keepCodegenPart(M, Split->PartOf, 0);
+ if (std::optional Failure =
+ generateCode(M, Machine, CodeGenFileType::ObjectFile,
+ Options.Outputs[0], Temporaries[0]))
+ Failures.set(std::move(*Failure));
+ return;
+ }
+ std::string LookupError;
+ std::unique_ptr PartMachine =
+ createTargetMachine(Options.Target, Options.OptLevel, LookupError);
+ if (!PartMachine)
+ return Failures.set({"target_machine_failed", LookupError});
+ LLVMContext Context;
+ std::unique_ptr Part;
+ if (Split) {
+ Expected> Parsed = parseBitcodeFile(
+ MemoryBufferRef(
+ StringRef(Split->Bitcode.data(), Split->Bitcode.size()), "part"),
+ Context);
+ if (!Parsed)
+ return Failures.set(
+ {"partition_failed", toString(Parsed.takeError())});
+ Part = std::move(*Parsed);
+ keepCodegenPart(*Part, Split->PartOf, I);
+ } else {
+ Part = std::make_unique(Identifier, Context);
+ Part->setTargetTriple(TargetTriple);
+ Part->setDataLayout(Layout);
+ }
+ if (std::optional Failure =
+ generateCode(*Part, *PartMachine, CodeGenFileType::ObjectFile,
+ Options.Outputs[I], Temporaries[I]))
+ return Failures.set(std::move(*Failure));
+ });
+ std::optional Failure = Failures.take();
+ for (size_t I = 0; I < Count && !Failure; ++I)
+ Failure = publish(Temporaries[I][0], Options.Outputs[I]);
+ if (Failure)
+ for (const std::vector> &Paths : Temporaries)
+ for (const SmallString<256> &Temporary : Paths)
+ sys::fs::remove(Temporary);
+ return Failure;
+}
+
} // namespace
int emit(const EmitOptions &Options) {
@@ -1089,7 +1338,14 @@ int emit(const EmitOptions &Options) {
if (std::optional Failure = inspectProfile(Options))
return reportError(Failure->Code, Failure->Message,
Options.DiagnosticFormat);
- if (Options.Outputs.size() > 1)
+ if (Options.SplitCodegen && (Options.Outputs.size() < 2 ||
+ Options.OptLevel == "0" ||
+ !Options.CacheDirectory.empty()))
+ return reportError("invalid_split",
+ "--split codegen requires several outputs, an "
+ "optimized build, and no cache directory",
+ Options.DiagnosticFormat);
+ if (Options.Outputs.size() > 1 && !Options.SplitCodegen)
return emitPartitions(Options);
SMDiagnostic ParseDiagnostic;
@@ -1147,6 +1403,16 @@ int emit(const EmitOptions &Options) {
CodeGenFileType Type = Options.FileType == "obj"
? CodeGenFileType::ObjectFile
: CodeGenFileType::AssemblyFile;
+ if (Options.SplitCodegen) {
+ ProfileMatch Match;
+ if (std::optional Failure =
+ emitSplitCodegen(Options, *Mod, *Machine, Level, Match))
+ return reportError(Failure->Code, Failure->Message,
+ Options.DiagnosticFormat);
+ if (!Options.ProfileUse.empty())
+ reportProfileMatch(Match);
+ return 0;
+ }
std::vector> Temporaries;
ProfileMatch Match;
std::optional Failure =
diff --git a/native/llvm-codegen/src/emit.h b/native/llvm-codegen/src/emit.h
index 39b8c2463..b62a9d01c 100644
--- a/native/llvm-codegen/src/emit.h
+++ b/native/llvm-codegen/src/emit.h
@@ -26,6 +26,9 @@ struct EmitOptions {
std::string ProfileGenerate;
// Indexed profile (.profdata) that guides optimization. Empty: none.
std::string ProfileUse;
+ // With several outputs: optimize the module whole and divide only its code
+ // generation among the outputs.
+ bool SplitCodegen = false;
};
std::optional parseEmitOptions(int Argc, char **Argv);
diff --git a/packages/compiler/src/backend/native-codegen-core.ts b/packages/compiler/src/backend/native-codegen-core.ts
index dd17466eb..d75dc2097 100644
--- a/packages/compiler/src/backend/native-codegen-core.ts
+++ b/packages/compiler/src/backend/native-codegen-core.ts
@@ -32,6 +32,25 @@ export function nativeProgramPartitions(llvmBytes: number): number {
return Math.min(MAX_PROGRAM_PARTITIONS, 2 ** Math.ceil(Math.log2(required)));
}
+/** LLVM text that an optimized one-partition program generates code for on
+ * one thread, the LLVM text per code generation part, and the most parts. */
+const SINGLE_CODEGEN_BYTES = 128 * 1024;
+const CODEGEN_PART_BYTES = 64 * 1024;
+const MAX_CODEGEN_PARTS = 8;
+
+/** Code generation parts for an optimized executable object that
+ * nativeProgramPartitions keeps as one partition. The helper optimizes the
+ * module whole, as for a single output, and then generates code for the parts
+ * concurrently (`--split codegen`), so the generated code does not change.
+ * Returns 1 for modules of at most SINGLE_CODEGEN_BYTES and for partitioned
+ * ones. The count is a power of two that depends only on the module's size,
+ * so artifacts never vary with the host. */
+export function nativeCodegenParts(llvmBytes: number): number {
+ if (llvmBytes <= SINGLE_CODEGEN_BYTES || llvmBytes > SINGLE_PARTITION_BYTES) return 1;
+ const required = Math.ceil(llvmBytes / CODEGEN_PART_BYTES);
+ return Math.min(MAX_CODEGEN_PARTS, 2 ** Math.ceil(Math.log2(required)));
+}
+
/** Output paths for each partition; the first is the requested path. */
export function nativePartitionPaths(path: string, count: number): string[] {
const dot = path.lastIndexOf(".");
diff --git a/packages/compiler/src/backend/native-codegen.test.ts b/packages/compiler/src/backend/native-codegen.test.ts
index 64961bcaa..8ef940c6a 100644
--- a/packages/compiler/src/backend/native-codegen.test.ts
+++ b/packages/compiler/src/backend/native-codegen.test.ts
@@ -8,7 +8,11 @@ import {
NativeCodegenError,
validateNativeCodegenVersion,
} from "./native-codegen.js";
-import { nativePartitionPaths, nativeProgramPartitions } from "./native-codegen-core.js";
+import {
+ nativeCodegenParts,
+ nativePartitionPaths,
+ nativeProgramPartitions,
+} from "./native-codegen-core.js";
import { nativeArtifactDependenciesStillMatch } from "./native-toolchain.js";
import { LINUX_X64_GNU_TARGET, MACOS_ARM64_TARGET, WASM32_WASI_TARGET } from "./targets.js";
import { compilerReleaseVersion } from "../library/sidecar.js";
@@ -409,6 +413,41 @@ test("program partitions depend only on module size", () => {
]);
});
+test("code generation parts depend only on module size", () => {
+ const kilobyte = 1024;
+ expect(nativeCodegenParts(128 * kilobyte)).toBe(1);
+ expect(nativeCodegenParts(128 * kilobyte + 1)).toBe(4);
+ expect(nativeCodegenParts(256 * kilobyte)).toBe(4);
+ expect(nativeCodegenParts(256 * kilobyte + 1)).toBe(8);
+ expect(nativeCodegenParts(1024 * kilobyte)).toBe(8);
+ // Partitioned modules already compile their partitions concurrently.
+ expect(nativeCodegenParts(1024 * kilobyte + 1)).toBe(1);
+});
+
+test("split code generation passes --split codegen and no partition cache", async () => {
+ const pkg = await fakePackage();
+ const output = join(pkg.root, "split", "program.o");
+ const artifact = await emitNativeArtifact({
+ ...request(pkg.root, pkg.packageJson, output),
+ partitions: 4,
+ splitCodegen: true,
+ });
+ expect(artifact.outputPaths).toEqual(nativePartitionPaths(output, 4));
+ const args = (await readFile(pkg.log, "utf8")).trim().split(" ");
+ expect(args.filter((arg) => arg === "--output")).toHaveLength(4);
+ expect(args.join(" ")).toContain("--split codegen");
+ expect(args).not.toContain("--cache-dir");
+
+ // Partitions of the same module are a distinct artifact.
+ await emitNativeArtifact({
+ ...request(pkg.root, pkg.packageJson, join(pkg.root, "partitions", "program.o")),
+ partitions: 4,
+ });
+ const calls = (await readFile(pkg.log, "utf8")).trim().split("\n");
+ expect(calls).toHaveLength(2);
+ expect(calls[1]).not.toContain("--split");
+});
+
test("partitioned objects are emitted, cached and restored together", async () => {
const pkg = await fakePackage();
const first = join(pkg.root, "first", "program.o");
diff --git a/packages/compiler/src/backend/native-codegen.ts b/packages/compiler/src/backend/native-codegen.ts
index 84c76ce91..ba04bd7d6 100644
--- a/packages/compiler/src/backend/native-codegen.ts
+++ b/packages/compiler/src/backend/native-codegen.ts
@@ -125,6 +125,9 @@ export interface NativeCodegenOptions {
cacheRoot?: string | null;
/** Program partitions, written to nativePartitionPaths(outputPath). */
partitions?: number;
+ /** With several partitions: optimize the module whole and divide only its
+ * code generation among them. */
+ splitCodegen?: boolean;
/** Profile-guided optimization: instrumentation or an indexed profile. */
profile?: NativeProfileOptions;
}
@@ -351,6 +354,7 @@ function cacheKey(
.update("\0")
.update(options.sourcePath)
.update((options.partitions ?? 1) > 1 ? `\0partitions:${options.partitions}` : "")
+ .update(options.splitCodegen === true ? "\0split:codegen" : "")
.update(options.profile === undefined ? "" : `\0${nativeProfileKey(options.profile)}`)
.digest("hex");
}
@@ -454,7 +458,7 @@ export async function emitNativeArtifact(
// edit recompiles only the partitions it affects. The directory names the
// exact helper build and target; the helper keys entries by content.
const partitionCache =
- root === null || outputs.length < 2
+ root === null || outputs.length < 2 || options.splitCodegen === true
? null
: join(
root,
@@ -494,6 +498,7 @@ export async function emitNativeArtifact(
"--source-path",
options.sourcePath,
...(partitionCache === null ? [] : ["--cache-dir", partitionCache]),
+ ...(options.splitCodegen === true && outputs.length > 1 ? ["--split", "codegen"] : []),
...nativeProfileHelperArgs(options.profile),
]);
for (const stage of stages) {
diff --git a/packages/compiler/src/backend/native-tools.ts b/packages/compiler/src/backend/native-tools.ts
index 9d6f2cf94..71ad26111 100644
--- a/packages/compiler/src/backend/native-tools.ts
+++ b/packages/compiler/src/backend/native-tools.ts
@@ -49,6 +49,9 @@ export function emitNativeObject(options: {
outputKind?: NativeCodegenOutputKind;
/** Program partitions, written to nativePartitionPaths(outputPath). */
partitions?: number;
+ /** With several partitions: optimize the module whole and divide only its
+ * code generation among them. */
+ splitCodegen?: boolean;
/** Private directory where the helper reuses partition artifacts. */
partitionCache?: string;
/** Profile-guided optimization: instrumentation or an indexed profile. */
@@ -73,9 +76,11 @@ export function emitNativeObject(options: {
"json",
"--source-path",
options.sourcePath,
- ...(options.partitionCache === undefined || outputs.length < 2
- ? []
- : ["--cache-dir", options.partitionCache]),
+ ...(options.splitCodegen === true && outputs.length > 1
+ ? ["--split", "codegen"]
+ : options.partitionCache === undefined || outputs.length < 2
+ ? []
+ : ["--cache-dir", options.partitionCache]),
...nativeProfileHelperArgs(options.profile),
]);
for (const output of outputs) requireNativeArtifact(output);
diff --git a/packages/compiler/src/index.ts b/packages/compiler/src/index.ts
index 079db8494..73162b488 100644
--- a/packages/compiler/src/index.ts
+++ b/packages/compiler/src/index.ts
@@ -95,7 +95,11 @@ import {
NativeCodegenError,
writeLlvmSource,
} from "./backend/native-codegen.js";
-import { nativePartitionPaths, nativeProgramPartitions } from "./backend/native-codegen-core.js";
+import {
+ nativeCodegenParts,
+ nativePartitionPaths,
+ nativeProgramPartitions,
+} from "./backend/native-codegen-core.js";
import { privateSiblingPath } from "./backend/build-cache.js";
import { nativeCodegenTarget, nativeCodegenTargetRefusal } from "./backend/targets.js";
import { windowsSubsystemLinkerArgs, type WindowsSubsystem } from "./backend/targets.js";
@@ -758,8 +762,11 @@ async function emitNativeProgramObject(
// The validated program-object artifact is a single object by contract.
const target = nativeCodegenTarget();
const llvmBytes = llvmInputBytes(llvm);
- const partitions =
- opts.nativeProgramObject === true || target === null ? 1 : nativeProgramPartitions(llvmBytes);
+ const single = opts.nativeProgramObject === true || target === null;
+ // An optimized program that stays one partition is optimized whole and
+ // generates code for its parts concurrently.
+ const codegenParts = single || opts.optimization === "dev" ? 1 : nativeCodegenParts(llvmBytes);
+ const partitions = single ? 1 : Math.max(nativeProgramPartitions(llvmBytes), codegenParts);
try {
const artifact = await emitNativeArtifact({
outputPath: linkPath,
@@ -769,6 +776,7 @@ async function emitNativeProgramObject(
optimization: opts.optimization === "dev" ? "0" : "2",
...(opts.sanitize === undefined ? {} : { sanitize: opts.sanitize }),
partitions,
+ ...(codegenParts > 1 ? { splitCodegen: true } : {}),
...(profile === undefined ? {} : { profile }),
});
return {
diff --git a/packages/compiler/src/native/compiler.ts b/packages/compiler/src/native/compiler.ts
index 942565327..1cf4f7eac 100644
--- a/packages/compiler/src/native/compiler.ts
+++ b/packages/compiler/src/native/compiler.ts
@@ -39,6 +39,7 @@ import {
import { RuntimePackError } from "../backend/runtime-pack-core.js";
import {
NativeCodegenError,
+ nativeCodegenParts,
nativePartitionPaths,
nativeProgramPartitions,
} from "../backend/native-codegen-core.js";
@@ -200,6 +201,7 @@ export class NativeCompiler {
partitions = 1,
profile?: NativeProfileOptions,
llvmDigest: () => string = () => fileDigest(input),
+ splitCodegen = false,
): NativeObjects {
const toolchain = this.toolchain;
const outputs = nativePartitionPaths(output, partitions);
@@ -227,6 +229,7 @@ export class NativeCompiler {
identity: readFileSync(join(toolchain.helperPackageRoot, "package.json"), "utf8"),
version: toolchain.compilerVersion,
...(partitions > 1 ? { partitions } : {}),
+ ...(splitCodegen ? { split: "codegen" } : {}),
...(profile === undefined ? {} : { profile: nativeProfileKey(profile) }),
}),
);
@@ -259,7 +262,7 @@ export class NativeCompiler {
}
}
const partitionCache =
- this.cache === null || identity === null || partitions < 2
+ this.cache === null || identity === null || partitions < 2 || splitCodegen
? null
: this.cache.partitionDirectory(identity);
const { profileMatch } = emitNativeObject({
@@ -274,6 +277,7 @@ export class NativeCompiler {
optimization,
outputKind,
partitions,
+ ...(splitCodegen ? { splitCodegen } : {}),
...(partitionCache === null ? {} : { partitionCache }),
...(profile === undefined ? {} : { profile }),
});
@@ -323,7 +327,11 @@ export class NativeCompiler {
: null;
const split = llvm === null ? null : splitLlvmLibraryProgram(llvm);
if (split === null) {
- const partitions = library ? 1 : nativeProgramPartitions(statSync(input).size);
+ const size = statSync(input).size;
+ // An optimized program that stays one partition is optimized whole and
+ // generates code for its parts concurrently.
+ const codegenParts = library || optimization === "dev" ? 1 : nativeCodegenParts(size);
+ const partitions = library ? 1 : Math.max(nativeProgramPartitions(size), codegenParts);
return this.emitObject(
input,
output,
@@ -333,6 +341,7 @@ export class NativeCompiler {
partitions,
profile,
llvmDigest,
+ codegenParts > 1,
);
}
const objects: string[] = [];
diff --git a/packages/compiler/test/native-partitions.test.ts b/packages/compiler/test/native-partitions.test.ts
index 113cd95ad..c86a37aa4 100644
--- a/packages/compiler/test/native-partitions.test.ts
+++ b/packages/compiler/test/native-partitions.test.ts
@@ -85,7 +85,13 @@ function expectedExit(edit = 0, count = 24): number {
async function emit(
dir: string,
source: string,
- options: { outputs: number; optLevel: "0" | "2"; cache?: string; name: string },
+ options: {
+ outputs: number;
+ optLevel: "0" | "2";
+ cache?: string;
+ split?: boolean;
+ name: string;
+ },
): Promise {
const input = join(dir, `${options.name}.ll`);
await writeFile(input, source);
@@ -110,6 +116,7 @@ async function emit(
"--source-path",
"/src/program.ts",
...(options.cache === undefined ? [] : ["--cache-dir", options.cache]),
+ ...(options.split === true ? ["--split", "codegen"] : []),
]);
return outputs;
}
@@ -162,6 +169,181 @@ describe.runIf(helper !== null)("LLVM program partitions", () => {
});
}
+ test("split code generation links deterministically and defines every function once", async () => {
+ const dir = await mkdtemp(join(tmpdir(), "scriptc-split-codegen-"));
+ dirs.push(dir);
+ const parts = await emit(dir, program(), {
+ outputs: 3,
+ optLevel: "2",
+ split: true,
+ name: "split",
+ });
+ expect(await exitCode(dir, parts, "split")).toBe(expectedExit());
+ const again = await emit(dir, program(), {
+ outputs: 3,
+ optLevel: "2",
+ split: true,
+ name: "again",
+ });
+ const contents = (paths: string[]) => Promise.all(paths.map((path) => readFile(path)));
+ expect(await contents(again)).toEqual(await contents(parts));
+
+ // The module is optimized whole, so the parts define the same functions as
+ // a single object, each in one part.
+ const defined = async (object: string) => {
+ const { stdout } = await execFileAsync("nm", ["--defined-only", object]);
+ return stdout
+ .split("\n")
+ .map((line) => /^[0-9a-fA-F]+ [Tt] _?(\w+)$/.exec(line.trim())?.[1])
+ .filter((name): name is string => name !== undefined);
+ };
+ const whole = await defined(
+ (await emit(dir, program(), { outputs: 1, optLevel: "2", name: "whole" }))[0]!,
+ );
+ const split = await Promise.all(parts.map(defined));
+ expect(split.flat().sort()).toEqual([...whole].sort());
+ for (const names of split) expect(names.length).toBeGreaterThan(0);
+ });
+
+ test("split code generation keeps local a definition that only its own part uses", async () => {
+ const dir = await mkdtemp(join(tmpdir(), "scriptc-split-codegen-local-"));
+ dirs.push(dir);
+ const count = 24;
+ const lines = ["@seed = internal global i64 5", "@total = internal global i64 0"];
+ for (let index = 0; index < count; index++)
+ lines.push(
+ `define internal i64 @leaf${index}(i64 %value) noinline {`,
+ ` %scaled = mul i64 %value, ${index + 3}`,
+ " ret i64 %scaled",
+ "}",
+ `define internal void @step${index}(i64 %value) noinline {`,
+ ` %scaled = call i64 @leaf${index}(i64 %value)`,
+ " %old = load i64, ptr @total",
+ " %new = add i64 %old, %scaled",
+ " store i64 %new, ptr @total",
+ " ret void",
+ "}",
+ );
+ lines.push(
+ "define i32 @main() {",
+ " %seed = load volatile i64, ptr @seed",
+ ...Array.from({ length: count }, (_, index) => ` call void @step${index}(i64 %seed)`),
+ " %total = load i64, ptr @total",
+ " %low = urem i64 %total, 251",
+ " %code = trunc i64 %low to i32",
+ " ret i32 %code",
+ "}",
+ );
+ const parts = await emit(dir, lines.join("\n") + "\n", {
+ outputs: 3,
+ optLevel: "2",
+ split: true,
+ name: "local",
+ });
+ let sum = 0;
+ for (let index = 0; index < count; index++) sum += 5 * (index + 3);
+ expect(await exitCode(dir, parts, "local")).toBe(sum % 251);
+ const symbols = await Promise.all(
+ parts.map(async (part) => (await execFileAsync("nm", ["--defined-only", part])).stdout),
+ );
+ const bindings = (listing: string, prefix: string) =>
+ [...listing.matchAll(new RegExp(`^[0-9a-fA-F]+ ([Tt]) _?${prefix}\\d+$`, "gm"))].map(
+ (match) => match[1],
+ );
+ // A step in another part than main's is called across parts. A leaf is
+ // called only by its step, which is in the same part unless a part
+ // boundary falls between them.
+ const mainPart = symbols.findIndex((listing) => / T _?main$/m.test(listing));
+ expect(mainPart).toBeGreaterThan(0);
+ const crossPart = symbols
+ .filter((_, index) => index !== mainPart)
+ .flatMap((listing) => bindings(listing, "step"));
+ expect(crossPart.length).toBeGreaterThan(0);
+ expect(new Set(crossPart)).toEqual(new Set(["T"]));
+ const leaves = symbols.flatMap((listing) => bindings(listing, "leaf"));
+ expect(leaves).toHaveLength(count);
+ expect(leaves.filter((bound) => bound === "t").length).toBeGreaterThanOrEqual(count - 2);
+ });
+
+ test("split code generation links a thread-local global that only one part uses", async () => {
+ const dir = await mkdtemp(join(tmpdir(), "scriptc-split-codegen-tls-"));
+ dirs.push(dir);
+ // @touch and main reach @slots through one constant expression, which
+ // outlives their bodies in the part that holds neither.
+ const thread = [
+ "@calls = internal thread_local(localexec) global i64 0",
+ "@slots = internal thread_local(localexec) global [2 x i64] zeroinitializer",
+ "define internal void @touch() noinline {",
+ " store volatile i64 1, ptr getelementptr inbounds (i8, ptr @slots, i64 8)",
+ " ret void",
+ "}",
+ ].join("\n");
+ const source = program(0, thread).replace(
+ " %total = load i64, ptr @total",
+ [
+ " store volatile i64 1, ptr @calls",
+ " call void @touch()",
+ " %slot = load volatile i64, ptr getelementptr inbounds (i8, ptr @slots, i64 8)",
+ " %total = load i64, ptr @total",
+ ].join("\n"),
+ );
+ const parts = await emit(dir, source, { outputs: 3, optLevel: "2", split: true, name: "tls" });
+ expect(await exitCode(dir, parts, "tls")).toBe(expectedExit());
+ });
+
+ test("split code generation keeps a module that one function dominates whole", async () => {
+ const dir = await mkdtemp(join(tmpdir(), "scriptc-split-codegen-whole-"));
+ dirs.push(dir);
+ const body = Array.from(
+ { length: 400 },
+ (_, index) => ` %v${index + 1} = mul i64 %v${index}, %v${index}`,
+ );
+ const heavy = [
+ "@sink = internal global i64 0",
+ "define internal void @heavy(i64 %v0) noinline {",
+ ...body,
+ " store volatile i64 %v400, ptr @sink",
+ " ret void",
+ "}",
+ ].join("\n");
+ const source = program(0, heavy, 2).replace(
+ " %total = load i64, ptr @total",
+ " call void @heavy(i64 %second)\n %total = load i64, ptr @total",
+ );
+ const parts = await emit(dir, source, {
+ outputs: 3,
+ optLevel: "2",
+ split: true,
+ name: "whole",
+ });
+ expect(await exitCode(dir, parts, "whole")).toBe(expectedExit(0, 2));
+ const { stdout } = await execFileAsync("nm", ["--defined-only", parts[0]!]);
+ expect(stdout).toMatch(/ t _?heavy$/m);
+ for (const part of parts.slice(1)) {
+ const listed = await execFileAsync("nm", ["--defined-only", part]);
+ expect(listed.stdout.trim()).toBe("");
+ }
+ // The first output is the object that a single output produces.
+ const first = await readFile(parts[0]!);
+ const [single] = await emit(dir, source, { outputs: 1, optLevel: "2", name: "whole" });
+ expect(await readFile(single!)).toEqual(first);
+ });
+
+ test("split code generation requires several optimized outputs and no cache", async () => {
+ const dir = await mkdtemp(join(tmpdir(), "scriptc-split-codegen-invalid-"));
+ dirs.push(dir);
+ for (const options of [
+ { outputs: 1, optLevel: "2" },
+ { outputs: 3, optLevel: "0" },
+ { outputs: 3, optLevel: "2", cache: join(dir, "cache") },
+ ] as const) {
+ const failure = emit(dir, program(), { ...options, split: true, name: "invalid" });
+ await expect(failure).rejects.toMatchObject({
+ stderr: expect.stringContaining("invalid_split"),
+ });
+ }
+ });
+
test("functions under one group line share a partition", async () => {
const dir = await mkdtemp(join(tmpdir(), "scriptc-partitions-groups-"));
dirs.push(dir);