diff --git a/docs/content/docs/cli.mdx b/docs/content/docs/cli.mdx index 734c07da6..f2d91f1bd 100644 --- a/docs/content/docs/cli.mdx +++ b/docs/content/docs/cli.mdx @@ -173,10 +173,10 @@ With `--sparse`, the profile leaves out the functions the runs never reached. Th
On x86-64 targets, compile the program's code twice, for the baseline and for x86-64-v3, and choose the copy when the program starts. The executable runs on any x86-64 CPU and uses the x86-64-v3 code on CPUs that support it. Not used with a --target-cpu level above baseline. Valid only for executable builds. See Runtime CPU dispatch.
--profile-generate <pattern>
-
On Linux, build an instrumented executable for profile-guided optimization. Each run writes a raw profile to the path pattern, where %p expands to the process ID; the LLVM_PROFILE_FILE environment variable overrides the pattern at run time. A relative pattern resolves from the current working directory. Instrumented programs run slower. Valid only for executable builds with --optimization=release or speed.
+
On Linux, build an instrumented executable for profile-guided optimization. Each run writes a raw profile to the path pattern, where %p expands to the process ID; the LLVM_PROFILE_FILE environment variable overrides the pattern at run time. A relative pattern resolves from the current working directory. Instrumented programs run slower. With --profile-use, the build is optimized by that profile and instrumented after inlining, the second phase of a context-sensitive profile. Valid only for executable builds with --optimization=release or speed.
--profile-use <file.profdata>
-
Optimize the executable with a profile from scriptc merge-profiles: code is laid out, inlined, and split by how the profiled runs used it. Code the profile does not describe, or describes from an older version of the program, is optimized as without a profile, so a stale profile only loses part of its gain. A new or changed profile compiles every partition of the program again. scriptc build prints how the profile matched the program's functions on stderr; see Profile-guided optimization. Valid only for executable builds with --optimization=release or speed.
+
Optimize the executable with a profile from scriptc merge-profiles: code is laid out, inlined, and split by how the profiled runs used it. Code the profile does not describe, or describes from an older version of the program, is optimized as without a profile, so a stale profile only loses part of its gain. A new or changed profile compiles every partition of the program again. scriptc build prints how the profile matched the program's functions on stderr; see Profile-guided optimization. A profile with context-sensitive counts also orders each function's blocks by them on Linux. Valid only for executable builds with --optimization=release or speed.
--npm-static <pkg[,pkg…]|auto>
Name npm packages to compile even if they don't ship their own type declarations (repeatable). A trailing ..., as in fs-extra..., also names every package that package depends on. auto uses the default automatic selection. Packages that can't compile produce diagnostics. See npm Dependencies.
@@ -287,6 +287,23 @@ scriptc: warning: 3739 of the 8947 functions app.profdata describes (41.8%) chan The build still succeeds. Collecting the profile again with the current program and scriptc version restores the counts. +### Context-sensitive profiles + +A profile from the steps above counts each function's branches before the optimizer inlines calls. A second phase counts them again after inlining, separately for every place a function was inlined into: + +```console +$ scriptc build app.ts --optimization=speed --profile-use app.profdata --profile-generate profiles-cs/%p.profraw -o app-instrumented-cs +$ ./app-instrumented-cs typical-input.json +$ scriptc merge-profiles -o app-cs.profdata app.profdata profiles-cs/*.profraw +$ scriptc build app.ts --optimization=speed --profile-use app-cs.profdata +``` + +- **First phase:** `--profile-generate` alone. It instruments the program before inlining and gives the profile of the steps above. +- **Second phase:** `--profile-use` with the first profile and `--profile-generate` together. The build is optimized by the first profile and instrumented after inlining, and `scriptc merge-profiles` adds its counts to the first profile. +- **Optimized build:** `--profile-use` with the merged profile. On Linux, it also orders each function's blocks by the counts and moves blocks the runs never reached into separate sections, so the code that runs sits together. + +A profile that already holds context-sensitive counts cannot start a second phase. Merge the second phase's raw profiles into the first-phase profile instead. Both phases run the same workloads. + ## Target CPU By default, x86-64 executables run on any x86-64 CPU, so the compiler uses only the baseline instruction set (SSE2). `--target-cpu` compiles the program's code for a higher x86-64 microarchitecture level. Each level includes the ones below it: diff --git a/native/llvm-codegen/src/emit.cpp b/native/llvm-codegen/src/emit.cpp index d3d868fae..98e07e9ce 100644 --- a/native/llvm-codegen/src/emit.cpp +++ b/native/llvm-codegen/src/emit.cpp @@ -111,9 +111,6 @@ std::optional parseEmitOptions(int Argc, char **Argv) { } if (Options.Input.empty() || Options.Outputs.empty()) return std::nullopt; - // A build either collects a profile or uses one. - if (!Options.ProfileGenerate.empty() && !Options.ProfileUse.empty()) - return std::nullopt; return Options; } @@ -150,20 +147,48 @@ PipelineTuningOptions tuningFor(OptimizationLevel Level) { return Tuning; } -// The profile-guided optimization a build asks for: IR instrumentation that -// writes raw profiles to the ProfileGenerate pattern, or annotation from the -// indexed ProfileUse profile. Functions the profile does not describe, or -// whose control flow changed since it was collected, are optimized as without -// one, so a stale profile loses only its gain. +// Whether the ProfileUse profile also holds context-sensitive counts, which +// a second instrumentation after inlining recorded (inspectProfile). +bool ContextProfile = false; + +// The profile-guided optimization a build asks for: +// - IR instrumentation that writes raw profiles to the ProfileGenerate +// pattern; +// - annotation from the indexed ProfileUse profile, and from its +// context-sensitive counts after inlining when it has them; +// - with both, annotation from ProfileUse and a second instrumentation after +// inlining that writes context-sensitive counts to the ProfileGenerate +// pattern. `scriptc merge-profiles` adds them to the first profile. +// Functions the profile does not describe, or whose control flow changed +// since it was collected, are optimized as without one, so a stale profile +// loses only its gain. std::optional profileOptions(const EmitOptions &Options) { + if (!Options.ProfileGenerate.empty() && !Options.ProfileUse.empty()) + return PGOOptions(Options.ProfileUse, Options.ProfileGenerate, "", "", + PGOOptions::IRUse, PGOOptions::CSIRInstr); if (!Options.ProfileGenerate.empty()) return PGOOptions(Options.ProfileGenerate, "", "", "", PGOOptions::IRInstr); if (!Options.ProfileUse.empty()) - return PGOOptions(Options.ProfileUse, "", "", "", PGOOptions::IRUse); + return PGOOptions(Options.ProfileUse, "", "", "", PGOOptions::IRUse, + ContextProfile ? PGOOptions::CSIRUse + : PGOOptions::NoCSAction); return std::nullopt; } +// Block placement by the extended TSP model, which needs the exact block +// counts a context-sensitive profile gives. The option is process-wide; the +// helper sets it once. +void enableExtTspPlacement() { + static const bool Enabled = [] { + if (cl::Option *Option = cl::getRegisteredOptions().lookup( + "enable-ext-tsp-block-placement")) + Option->addOccurrence(0, "enable-ext-tsp-block-placement", "true"); + return true; + }(); + (void)Enabled; +} + // With a profile, the thin link imports the callees of hot call edges up to // 10 times, and of the hottest edges up to 100 times, the usual size limit, // into every partition that calls them. On large programs that triples the @@ -318,7 +343,8 @@ void reportProfileMatch(const ProfileMatch &Match) { } // Reads the profile's header once, before any partition needs it: it must -// hold IR-level counts. +// hold IR-level counts. Context-sensitive counts lay out the code by them; +// a profile that has them cannot take a second set. std::optional inspectProfile(const EmitOptions &Options) { if (Options.ProfileUse.empty()) return std::nullopt; @@ -333,6 +359,16 @@ std::optional inspectProfile(const EmitOptions &Options) { return EmitFailure{"profile_read_failed", Options.ProfileUse + ": not an IR-level instrumentation profile"}; + ContextProfile = (*Reader)->hasCSIRLevelProfile(); + if (ContextProfile && !Options.ProfileGenerate.empty()) + return EmitFailure{"profile_read_failed", + Options.ProfileUse + + ": already holds context-sensitive counts; " + "collect them on top of the IR-level profile"}; + if (ContextProfile && Triple(Options.Target).isOSBinFormatELF()) { + enableProfileLayout(); + enableExtTspPlacement(); + } return std::nullopt; } @@ -804,6 +840,14 @@ compileOptimizedPartitions(const EmitOptions &Options, : Level == OptimizationLevel::O3 ? 3 : 2; Config.PTO = tuningFor(Level); + // The thin backend instruments or annotates context-sensitive counts after + // its inlining (see profileOptions). + if (!Options.ProfileGenerate.empty() && !Options.ProfileUse.empty()) { + Config.RunCSIRInstr = true; + Config.CSIRProfile = Options.ProfileGenerate; + } else if (ContextProfile) { + Config.CSIRProfile = Options.ProfileUse; + } Config.CGFileType = CodeGenFileType::ObjectFile; Config.DefaultTriple = Options.Target; Config.DiagHandler = [&](const DiagnosticInfo &Info) { diff --git a/native/llvm-codegen/src/emit.h b/native/llvm-codegen/src/emit.h index b62a9d01c..1bdc578df 100644 --- a/native/llvm-codegen/src/emit.h +++ b/native/llvm-codegen/src/emit.h @@ -22,7 +22,9 @@ struct EmitOptions { std::string CacheDirectory; // Instruments the program for IR profiling. The instrumented executable // writes raw profiles to this path pattern (LLVM_PROFILE_FILE overrides it - // at run time). Empty: no instrumentation. + // at run time). Empty: no instrumentation. With ProfileUse, the program is + // optimized by that profile and instrumented again after inlining, which + // records context-sensitive counts. std::string ProfileGenerate; // Indexed profile (.profdata) that guides optimization. Empty: none. std::string ProfileUse; diff --git a/native/llvm-codegen/src/target.cpp b/native/llvm-codegen/src/target.cpp index fd2866761..2695d478f 100644 --- a/native/llvm-codegen/src/target.cpp +++ b/native/llvm-codegen/src/target.cpp @@ -118,11 +118,19 @@ CodeGenOptLevel codeGenLevel(StringRef Level) { return CodeGenOptLevel::Default; } +static bool ProfileLayout = false; + +void enableProfileLayout() { ProfileLayout = true; } + TargetOptions targetOptions(const Triple &TargetTriple) { TargetOptions Options; // The Android API 26 runtime pack uses the NDK's emulated TLS ABI. // Program globals and runtime thread instances must use the same ABI. Options.EmulatedTLS = TargetTriple.isAndroid(); + if (ProfileLayout && TargetTriple.isOSBinFormatELF()) { + Options.FunctionSections = true; + Options.EnableMachineFunctionSplitter = true; + } return Options; } diff --git a/native/llvm-codegen/src/target.h b/native/llvm-codegen/src/target.h index 5049d34f5..5ad469bcf 100644 --- a/native/llvm-codegen/src/target.h +++ b/native/llvm-codegen/src/target.h @@ -40,6 +40,10 @@ bool supportsTarget(llvm::StringRef Triple); // The target options and code generation level every helper target machine // uses, shared with the partitioned optimizer's configuration. llvm::TargetOptions targetOptions(const llvm::Triple &TargetTriple); +// Lays out code by a profile's context-sensitive counts from now on: one +// section per function, and cold blocks split into .text.split sections. +// ELF targets only; the helper calls it once, before any target machine. +void enableProfileLayout(); llvm::CodeGenOptLevel codeGenLevel(llvm::StringRef OptLevel); std::unique_ptr createTargetMachine(llvm::StringRef Triple, llvm::StringRef OptLevel, diff --git a/packages/cli/test/output-options.test.ts b/packages/cli/test/output-options.test.ts index 45d5c55b8..d1e157ea7 100644 --- a/packages/cli/test/output-options.test.ts +++ b/packages/cli/test/output-options.test.ts @@ -55,7 +55,6 @@ describe("output option compatibility", () => { [{ emit: "asm", cpuDispatch: true }, /cpu-dispatch/], [{ emit: "obj", profileGenerate: "/tmp/%p.profraw" }, /profile-generate/], [{ emit: "llvm", profileUse: "/tmp/a.profdata" }, /profile-use/], - [{ profileGenerate: "/tmp/%p.profraw", profileUse: "/tmp/a.profdata" }, /cannot be combined/], ] as const)("rejects %j", (override, message) => { const result = resolveOutputOptions("build", { ...BASE, ...override }); expect(result).toEqual({ ok: false, message: expect.stringMatching(message) }); @@ -120,4 +119,14 @@ describe("output option compatibility", () => { outputKind: "exe", }); }); + + test("a second profile phase combines --profile-use and --profile-generate", () => { + expect( + resolveOutputOptions("build", { + ...BASE, + profileGenerate: "/tmp/%p.profraw", + profileUse: "/tmp/a.profdata", + }), + ).toMatchObject({ ok: true, outputKind: "exe" }); + }); }); diff --git a/packages/cli/test/profile.test.ts b/packages/cli/test/profile.test.ts index da40a15b6..ab3f198ac 100644 --- a/packages/cli/test/profile.test.ts +++ b/packages/cli/test/profile.test.ts @@ -109,10 +109,6 @@ test("profile options are refused where they cannot apply", async () => { await writeFile(entry, PROGRAM); for (const [args, message] of [ [["build", entry, "--emit=llvm", "--profile-use", "a.profdata"], "--emit=exe"], - [ - ["build", entry, "--profile-generate", "%p.profraw", "--profile-use", "a.profdata"], - "cannot be combined", - ], [["coverage", entry, "--profile-use", "a.profdata"], "build/run option"], [["merge-profiles", "a.profraw"], "merge-profiles takes -o"], [["merge-profiles", "-o", "out.profdata"], "merge-profiles takes -o"], diff --git a/packages/compiler/src/backend/link-plan-core.ts b/packages/compiler/src/backend/link-plan-core.ts index f810a4a17..847dca065 100644 --- a/packages/compiler/src/backend/link-plan-core.ts +++ b/packages/compiler/src/backend/link-plan-core.ts @@ -8,7 +8,7 @@ import { type NativeTargetSpec, type WindowsSubsystem, } from "./targets.js"; -import { profileGenerateLinkerArgs } from "./native-profile.js"; +import { profileGenerateLinkerArgs, profileLayoutLinkerArgs } from "./native-profile.js"; export interface ExecutableLinkInputs { inputs: string[]; @@ -34,6 +34,8 @@ export function executableLinkInputs(options: { windowsSubsystem?: WindowsSubsystem; /** Instrumented for profile-guided optimization. */ profileGenerate?: boolean; + /** Laid out by a profile's context-sensitive counts (nativeProfileLayout). */ + profileLayout?: boolean; }): ExecutableLinkInputs { if (options.ffiFrameworks.length !== 0 && options.target.platform !== "darwin") throw new Error("FFI frameworks require a Darwin target"); @@ -71,6 +73,7 @@ export function executableLinkInputs(options: { ...executableStaticLinkerArgs(options.target, options.staticLink ?? false), ...windowsSubsystemLinkerArgs(options.target.platform, options.windowsSubsystem), ...profileGenerateLinkerArgs(options.profileGenerate ?? false), + ...profileLayoutLinkerArgs(options.target, options.profileLayout ?? false), ], }; } diff --git a/packages/compiler/src/backend/link-plan.ts b/packages/compiler/src/backend/link-plan.ts index 73139d22a..c3c0cb5a7 100644 --- a/packages/compiler/src/backend/link-plan.ts +++ b/packages/compiler/src/backend/link-plan.ts @@ -44,6 +44,7 @@ export async function createNativeLinkPlan(options: { staticLink?: boolean; windowsSubsystem?: WindowsSubsystem; profileGenerate?: boolean; + profileLayout?: boolean; programObjectDependencies?: readonly NativeArtifactDependency[]; env?: NodeJS.ProcessEnv; resolver?: (specifier: string) => string; @@ -78,6 +79,7 @@ export async function createNativeLinkPlan(options: { ? {} : { windowsSubsystem: options.windowsSubsystem }), ...(options.profileGenerate === true ? { profileGenerate: true } : {}), + ...(options.profileLayout === true ? { profileLayout: true } : {}), }), dependencyPaths: [...runtimePack.dependencyPaths, ...(options.ffi?.libraries ?? [])], programObjectDependencies: [...(options.programObjectDependencies ?? [])], diff --git a/packages/compiler/src/backend/native-codegen.test.ts b/packages/compiler/src/backend/native-codegen.test.ts index 8ef940c6a..ff0224746 100644 --- a/packages/compiler/src/backend/native-codegen.test.ts +++ b/packages/compiler/src/backend/native-codegen.test.ts @@ -186,13 +186,17 @@ test("profiles reach the helper and key the native cache by their bytes", async const used = { ...base, outputPath: join(pkg.root, "used.o"), - profile: { use: { path: "/profiles/a.profdata", digest: "a".repeat(64) } }, + profile: { + use: { path: "/profiles/a.profdata", digest: "a".repeat(64), contextSensitive: false }, + }, }; await emitNativeArtifact(used); await emitNativeArtifact({ ...used, outputPath: join(pkg.root, "changed.o"), - profile: { use: { path: "/profiles/a.profdata", digest: "b".repeat(64) } }, + profile: { + use: { path: "/profiles/a.profdata", digest: "b".repeat(64), contextSensitive: false }, + }, }); await emitNativeArtifact({ ...used, outputPath: join(pkg.root, "again.o") }); const calls = (await readFile(pkg.log, "utf8")).trim().split("\n"); @@ -209,7 +213,9 @@ test("a cached artifact keeps the profile match of the build that produced it", }); const used = { ...request(pkg.root, pkg.packageJson, join(pkg.root, "used.o")), - profile: { use: { path: "/profiles/a.profdata", digest: "a".repeat(64) } }, + profile: { + use: { path: "/profiles/a.profdata", digest: "a".repeat(64), contextSensitive: false }, + }, }; const match = { annotated: 120, mismatched: 7, missing: 40, unknownPartitions: 0 }; expect((await emitNativeArtifact(used)).profileMatch).toEqual(match); diff --git a/packages/compiler/src/backend/native-profile.test.ts b/packages/compiler/src/backend/native-profile.test.ts index 18f0c6f73..4fffcf457 100644 --- a/packages/compiler/src/backend/native-profile.test.ts +++ b/packages/compiler/src/backend/native-profile.test.ts @@ -5,8 +5,11 @@ import { nativeProfileHelperArgs, renderNativeProfileMatch, nativeProfileKey, + nativeProfileLayout, nativeProfileRefusal, profileGenerateLinkerArgs, + profileHasContextCounts, + profileLayoutLinkerArgs, } from "./native-profile.js"; import { NATIVE_TARGETS } from "./targets.js"; @@ -27,10 +30,21 @@ test("profiles are accepted for optimized executables", () => { expect( nativeProfileRefusal({ ...BASE, platform: "darwin", profileUse: "/p/a.profdata" }), ).toBeNull(); + // The second phase of a context-sensitive profile. + expect( + nativeProfileRefusal({ + ...BASE, + profileGenerate: "/p/%p.profraw", + profileUse: "/p/a.profdata", + }), + ).toBeNull(); }); test.each([ - [{ profileGenerate: "/p/%p.profraw", profileUse: "/p/a.profdata" }, /cannot be combined/], + [ + { profileGenerate: "/p/%p.profraw", profileUse: "/p/a.profdata", platform: "darwin" }, + /Linux executables/, + ], [{ profileUse: "/p/a.profdata", outputKind: "obj" }, /--profile-use is only supported/], [{ profileGenerate: "/p/%p.profraw", sanitize: true }, /--sanitize/], [{ profileUse: "/p/a.profdata", optimization: "dev" }, /optimized build/], @@ -48,9 +62,47 @@ test("helper arguments and cache keys name the profile", () => { "/p/%p.profraw", ]); expect(nativeProfileKey({ generate: "/p/%p.profraw" })).toBe("profile-generate:/p/%p.profraw"); - const use = { use: { path: "/p/a.profdata", digest: "d".repeat(64) } }; + const use = { use: { path: "/p/a.profdata", digest: "d".repeat(64), contextSensitive: false } }; expect(nativeProfileHelperArgs(use)).toEqual(["--profile-use", "/p/a.profdata"]); expect(nativeProfileKey(use)).toBe(`profile-use:${"d".repeat(64)}`); + const both = { generate: "/p/%p.profraw", ...use }; + expect(nativeProfileHelperArgs(both)).toEqual([ + "--profile-generate", + "/p/%p.profraw", + "--profile-use", + "/p/a.profdata", + ]); + expect(nativeProfileKey(both)).toBe( + `profile-generate:/p/%p.profraw\0profile-use:${"d".repeat(64)}`, + ); +}); + +test("a profile's context-sensitive flag is read from its header", () => { + const header = (last: number) => + Uint8Array.from([0xff, 0x6c, 0x70, 0x72, 0x6f, 0x66, 0x69, 0x81, 12, 0, 0, 0, 0, 0, 0, last]); + // VARIANT_MASK_IR_PROF (bit 56) alone, then with VARIANT_MASK_CSIR_PROF (bit 57). + expect(profileHasContextCounts(header(0x01))).toBe(false); + expect(profileHasContextCounts(header(0x03))).toBe(true); + expect(profileHasContextCounts(header(0x03).slice(0, 15))).toBe(false); + const raw = header(0x03); + raw[0] = 0x00; + expect(profileHasContextCounts(raw)).toBe(false); +}); + +test("only a context-sensitive profile lays out code and keeps its section kinds apart", () => { + const use = (contextSensitive: boolean) => ({ + use: { path: "/p/a.profdata", digest: "d".repeat(64), contextSensitive }, + }); + expect(nativeProfileLayout(undefined)).toBe(false); + expect(nativeProfileLayout(use(false))).toBe(false); + expect(nativeProfileLayout(use(true))).toBe(true); + expect(nativeProfileLayout({ generate: "/p/%p.profraw", ...use(true) })).toBe(false); + const linux = { platform: "linux", defaultLinker: "clang" }; + expect(profileLayoutLinkerArgs(linux, true)).toEqual(["-Wl,-z,keep-text-section-prefix"]); + expect(profileLayoutLinkerArgs(linux, false)).toEqual([]); + expect(profileLayoutLinkerArgs({ platform: "darwin", defaultLinker: "clang" }, true)).toEqual([]); + // Zig links cross-compiled and glibc-versioned Linux targets and rejects the flag. + expect(profileLayoutLinkerArgs({ platform: "linux", defaultLinker: "zig" }, true)).toEqual([]); }); test("only the instrumented link adds the profile runtime", () => { diff --git a/packages/compiler/src/backend/native-profile.ts b/packages/compiler/src/backend/native-profile.ts index 85b1fdca4..3968ac712 100644 --- a/packages/compiler/src/backend/native-profile.ts +++ b/packages/compiler/src/backend/native-profile.ts @@ -5,14 +5,32 @@ * merge-profiles` turns those raw profiles into one indexed profile, and an * optimized build (`--profile-use`) lays out, inlines, and splits code by it. * The helper applies both in the pre-link pipeline of every partition. + * + * A second phase records context-sensitive counts: a build with both flags + * is optimized by the first profile and instrumented again after inlining, + * and merging its raw profiles into the first profile adds the counts of the + * inlined code. A profile with those counts also orders each function's + * blocks by them and moves cold blocks to separate sections. */ import type { NativeOptimization } from "./optimization.js"; export interface NativeProfileOptions { - /** Raw profile path pattern written by the instrumented executable. */ + /** Raw profile path pattern written by the instrumented executable. With + * `use`, the instrumentation runs after inlining and records + * context-sensitive counts. */ generate?: string; - /** Indexed profile and the digest of its bytes, which key every cache. */ - use?: { path: string; digest: string }; + /** Indexed profile, the digest of its bytes (which key every cache), and + * whether it holds context-sensitive counts. */ + use?: { path: string; digest: string; contextSensitive: boolean }; +} + +/** Whether an indexed profile holds context-sensitive counts: the header's + * version word carries LLVM's VARIANT_MASK_CSIR_PROF flag, bit 57, which is + * bit 1 of its last byte. */ +export function profileHasContextCounts(bytes: Uint8Array): boolean { + const magic = [0xff, 0x6c, 0x70, 0x72, 0x6f, 0x66, 0x69, 0x81]; + if (bytes.length < 16 || magic.some((byte, index) => bytes[index] !== byte)) return false; + return (bytes[15]! & 0x02) !== 0; } /** Why a request cannot use profiles, or null when it can. */ @@ -28,7 +46,6 @@ export function nativeProfileRefusal(options: { const use = options.profileUse !== undefined; if (!generate && !use) return null; const flag = generate ? "--profile-generate" : "--profile-use"; - if (generate && use) return "--profile-generate and --profile-use cannot be combined"; if (options.outputKind !== "exe") return `${flag} is only supported for executable output`; if (options.sanitize) return `${flag} cannot be combined with --sanitize`; if (options.optimization === "dev") @@ -39,9 +56,10 @@ export function nativeProfileRefusal(options: { } export function nativeProfileHelperArgs(profile: NativeProfileOptions | undefined): string[] { - if (profile?.generate !== undefined) return ["--profile-generate", profile.generate]; - if (profile?.use !== undefined) return ["--profile-use", profile.use.path]; - return []; + return [ + ...(profile?.generate === undefined ? [] : ["--profile-generate", profile.generate]), + ...(profile?.use === undefined ? [] : ["--profile-use", profile.use.path]), + ]; } /** How an optimized build's profile matched the program's functions, as the @@ -111,12 +129,32 @@ export function renderNativeProfileMatch(profile: string, match: NativeProfileMa /** Cache-key part; empty without a profile so other keys are unchanged. */ export function nativeProfileKey(profile: NativeProfileOptions | undefined): string { - if (profile?.generate !== undefined) return `profile-generate:${profile.generate}`; - if (profile?.use !== undefined) return `profile-use:${profile.use.digest}`; - return ""; + return [ + ...(profile?.generate === undefined ? [] : [`profile-generate:${profile.generate}`]), + ...(profile?.use === undefined ? [] : [`profile-use:${profile.use.digest}`]), + ].join("\0"); +} + +/** Whether a build lays out its code by context-sensitive counts: it uses a + * profile that has them, and records none. */ +export function nativeProfileLayout(profile: NativeProfileOptions | undefined): boolean { + return profile?.use?.contextSensitive === true && profile.generate === undefined; } /** The instrumented link adds the compiler driver's profile runtime. */ export function profileGenerateLinkerArgs(generate: boolean): string[] { return generate ? ["-fprofile-generate"] : []; } + +/** A link that lays out code by a profile keeps the hot, unlikely and split + * sections the helper writes apart, so each kind of code sits together. Zig, + * which links cross-compiled and glibc-versioned Linux targets, rejects the + * flag, so those links leave it out. */ +export function profileLayoutLinkerArgs( + target: { platform: string; defaultLinker: string }, + layout: boolean, +): string[] { + return layout && target.platform === "linux" && target.defaultLinker !== "zig" + ? ["-Wl,-z,keep-text-section-prefix"] + : []; +} diff --git a/packages/compiler/src/cli/output-options.ts b/packages/compiler/src/cli/output-options.ts index 207878692..53361a9a1 100644 --- a/packages/compiler/src/cli/output-options.ts +++ b/packages/compiler/src/cli/output-options.ts @@ -87,9 +87,6 @@ export function resolveOutputOptions( if (values.cpuDispatch && emit !== "exe") { return { ok: false, message: `--cpu-dispatch is only supported with --emit=exe` }; } - if (values.profileGenerate !== undefined && values.profileUse !== undefined) { - return { ok: false, message: `--profile-generate and --profile-use cannot be combined` }; - } if (values.profileGenerate !== undefined && emit !== "exe") { return { ok: false, message: `--profile-generate is only supported with --emit=exe` }; } diff --git a/packages/compiler/src/index.ts b/packages/compiler/src/index.ts index e81e4353a..f492ca672 100644 --- a/packages/compiler/src/index.ts +++ b/packages/compiler/src/index.ts @@ -112,7 +112,9 @@ import { RuntimePackError } from "./backend/runtime-pack.js"; import { createNativeLinkPlan } from "./backend/link-plan.js"; import { nativeProfileKey, + nativeProfileLayout, nativeProfileRefusal, + profileHasContextCounts, type NativeProfileMatch, type NativeProfileOptions, } from "./backend/native-profile.js"; @@ -619,6 +621,7 @@ async function compileExecutableNative( /** The kept LLVM text's shared name, when `llvmPath` is the build's private copy of it * (BuildSources): the spelling clang sees and native cache keys name. */ llvmVisiblePath?: string, + profileLayout = false, ): Promise { const programIsObject = /\.(?:o|obj)$/.test(llvmPath); const runtimePackTarget = programIsObject && !sanitize ? nativeCodegenTarget() : null; @@ -635,6 +638,7 @@ async function compileExecutableNative( ...(staticLink ? { staticLink: true } : {}), ...(windowsSubsystem === undefined ? {} : { windowsSubsystem }), ...(profileGenerate ? { profileGenerate: true } : {}), + ...(profileLayout ? { profileLayout: true } : {}), programObjectDependencies, }); const cacheableLinker = @@ -819,8 +823,8 @@ async function removeNativeProgramObject( async function readNativeProfile( opts: CompileRequestOptions, ): Promise { - if (opts.profileGenerate !== undefined) return { generate: opts.profileGenerate }; - if (opts.profileUse === undefined) return undefined; + const generate = opts.profileGenerate; + if (opts.profileUse === undefined) return generate === undefined ? undefined : { generate }; const path = resolve(opts.profileUse); let bytes: Buffer; try { @@ -830,7 +834,12 @@ async function readNativeProfile( `cannot read --profile-use file ${path}: ${err instanceof Error ? err.message : String(err)}`, ); } - return { use: { path, digest: createHash("sha256").update(bytes).digest("hex") } }; + const use = { + path, + digest: createHash("sha256").update(bytes).digest("hex"), + contextSensitive: profileHasContextCounts(bytes), + }; + return generate === undefined ? { use } : { generate, use }; } /** The digest of the request's function layout file, which keys the @@ -1879,6 +1888,7 @@ async function compileTracked( nativeProgramObject?.partitionPaths, profile?.generate !== undefined, hitSources.llvmPath, + nativeProfileLayout(profile), ); if (nativeProgramObject !== null && opts.nativeProgramObject === true) { await rename(nativeProgramObject.linkPath, nativeProgramObject.artifactPath); @@ -2026,6 +2036,7 @@ async function compileTracked( nativeProgramObject?.partitionPaths, profile?.generate !== undefined, llvmPath, + nativeProfileLayout(profile), ); timing("native-link"); if (nativeProgramObject !== null && opts.nativeProgramObject === true) { diff --git a/packages/compiler/src/native/compiler.ts b/packages/compiler/src/native/compiler.ts index 0dec85cbd..b46778827 100644 --- a/packages/compiler/src/native/compiler.ts +++ b/packages/compiler/src/native/compiler.ts @@ -64,8 +64,10 @@ import { evaluateNativeComptime } from "./comptime.js"; import { contentDigest, fileDigest, partsDigest, type NativeCache } from "./cache.js"; import { nativeProfileKey, + nativeProfileLayout, nativeProfileRefusal, parseNativeProfileMatch, + profileHasContextCounts, type NativeProfileMatch, type NativeProfileOptions, } from "../backend/native-profile.js"; @@ -113,11 +115,17 @@ function failure(error: unknown, entry: string, sources: Map): C /** The request's profile. An indexed profile is read here so every cache keys * its bytes. */ function nativeProfile(options: CompileRequestOptions): NativeProfileOptions | undefined { - if (options.profileGenerate !== undefined) return { generate: options.profileGenerate }; - if (options.profileUse === undefined) return undefined; + const generate = options.profileGenerate; + if (options.profileUse === undefined) return generate === undefined ? undefined : { generate }; const path = resolve(options.profileUse); try { - return { use: { path, digest: contentDigest(readFileSync(path)) } }; + const bytes = readFileSync(path); + const use = { + path, + digest: contentDigest(bytes), + contextSensitive: profileHasContextCounts(bytes), + }; + return generate === undefined ? { use } : { generate, use }; } catch (error) { throw new NativeCodegenError( "SC3002", @@ -625,6 +633,7 @@ export class NativeCompiler { ? {} : { windowsSubsystem: options.windowsSubsystem }), ...(profile?.generate === undefined ? {} : { profileGenerate: true }), + ...(nativeProfileLayout(profile) ? { profileLayout: true } : {}), }); // Linux links with lld when the driver finds it. const lld = options.sanitize ? [] : lldLinkerArgs(target.platform, toolchain.linker); diff --git a/packages/compiler/test/native-profile.test.ts b/packages/compiler/test/native-profile.test.ts index 9d04739d0..8abca4393 100644 --- a/packages/compiler/test/native-profile.test.ts +++ b/packages/compiler/test/native-profile.test.ts @@ -272,18 +272,54 @@ async function cacheEntries(cache: string): Promise> { } describe.runIf(helper !== null)("LLVM profile-guided optimization", () => { - test("a profile and instrumentation together are a usage error", async () => { - const dir = await mkdtemp(join(tmpdir(), "scriptc-profile-usage-")); - dirs.push(dir); - await expect( - emit(dir, program(), { - outputs: 1, - name: "both", - generate: join(dir, "%p.profraw"), - use: join(dir, "missing.profdata"), - }), - ).rejects.toMatchObject({ stderr: expect.stringContaining('"code":"usage"') }); - }); + test.runIf(profileLinker !== null)( + "a second phase records context-sensitive counts that lay out the code", + async () => { + const dir = await mkdtemp(join(tmpdir(), "scriptc-profile-context-")); + dirs.push(dir); + const first = await collect(dir, inlinedProgram(), 2, "first", inlinedExit); + // The second phase: optimized by the first profile, instrumented after inlining. + const raw = join(dir, "context-raw"); + const instrumented = await emit(dir, inlinedProgram(), { + outputs: 2, + name: "context", + generate: join(raw, "%p.profraw"), + use: first.profile, + }); + const executable = await link(dir, instrumented, "context", true); + expect(await run(executable, 1)).toBe(inlinedExit(1)); + expect(await run(executable, 3)).toBe(inlinedExit(3)); + const raws = (await readdir(raw)).map((file) => join(raw, file)); + expect(raws).toHaveLength(2); + const merged = join(dir, "context.profdata"); + await merge(merged, [first.profile, ...raws]); + // VARIANT_MASK_CSIR_PROF, bit 57 of the header's version word. + expect((await readFile(merged))[15]! & 0x02).toBe(0x02); + expect((await readFile(first.profile))[15]! & 0x02).toBe(0); + // A build with the merged profile gives every function its own section. + const objects = await emit(dir, inlinedProgram(), { + outputs: 2, + name: "laidout", + use: merged, + }); + const bytes = Buffer.concat(await Promise.all(objects.map((object) => readFile(object)))); + expect(bytes.includes(".text.hot.main")).toBe(true); + const optimized = await link(dir, objects, "laidout"); + expect(await run(optimized, 1)).toBe(inlinedExit(1)); + expect(await run(optimized, 3)).toBe(inlinedExit(3)); + // A profile that has context-sensitive counts cannot start another second phase. + await expect( + emit(dir, inlinedProgram(), { + outputs: 1, + name: "again", + generate: join(dir, "again-raw", "%p.profraw"), + use: merged, + }), + ).rejects.toMatchObject({ + stderr: expect.stringContaining("already holds context-sensitive counts"), + }); + }, + ); test("merge-profiles reports an input that is not a profile", async () => { const dir = await mkdtemp(join(tmpdir(), "scriptc-profile-merge-error-"));