Part 2: Profiling, PGO & LTO
Measuring PGO and LTO on a C++ SPH simulation with perf and flamegraphs, and why they barely helped.
A tested walkthrough for enabling Profile Guided Optimization and Link Time Optimization in a Meson project built with Clang, including the llvm-profdata merge step most guides forget.
I covered why PGO and LTO barely moved the needle on my SPH simulation in part 2 of my optimization series. This post skips the “does it actually help” question and just gives you the commands, because half the guides I found online gloss over a step that will make your build fail with a cryptic clang error.
PGO (Profile Guided Optimization) runs an instrumented build of your program on real input, records which branches and functions actually get hit, then feeds that back into a second build so the compiler can lay out hot/cold paths and inline based on evidence instead of guesses.
LTO (Link Time Optimization) lets the compiler optimize across translation units instead of one .cpp file at a time, so it can inline across files, drop dead code, and see through calls that would otherwise be opaque at the object-file level.
They’re orthogonal. You can use either alone, or stack both for the final shipping build.
b_lto_mode).lld is not mandatory, but it makes thin LTO caching simpler. GNU ld and gold both support LTO through the plugin interface, it’s just more moving parts.meson setup build --buildtype=release -Db_lto=true
meson compile -C buildThat’s full LTO: every translation unit gets bundled into one giant module at link time. It’s thorough, and it’s slow, both in link time and in RAM used by the linker. On a big project this can hurt.
Thin LTO fixes that by keeping per-module optimization but sharing summaries across modules, so the linker can parallelize the codegen step:
meson setup build --buildtype=release -Db_lto=true -Db_lto_mode=thin -Db_lto_threads=8
meson compile -C buildIf you want to confirm meson is actually passing the flag through and not silently ignoring it, -v your ninja build:
ninja -C build -v | grep -o '\-flto[^ ]*'
-flto=thinWant lld specifically? Set it before meson setup, not as a -D option:
CXX_LD=lld meson setup build --buildtype=release -Db_lto=true -Db_lto_mode=thinThis is the two-pass part.
meson setup build-pgo --buildtype=release -Db_pgo=generate
meson compile -C build-pgo
./build-pgo/your-binary --some-representative-workloadRunning the instrumented binary drops a file next to wherever you ran it from, named something like default_15853201381332895877_0.profraw. Not in the build directory. Wherever your current shell was when you launched the binary.
Meson does not merge the profile for you. There is no profdata or profraw string anywhere in mesonbuild’s source. If you just reconfigure with -Db_pgo=use and compile, you get this:
clang: error: Error in reading profile default.profdata: No such file or directory
ninja: build stopped: subcommand failed.Clang wants an already-merged, indexed default.profdata, and it looks for it in its own working directory, which for the ninja backend is the root of your build directory. So merge it there yourself:
llvm-profdata merge -output=build-pgo/default.profdata *.profrawThen reconfigure and rebuild:
meson configure build-pgo -Db_pgo=use
meson compile -C build-pgoIf you have several representative workloads, run the instrumented binary once per workload and hand all the resulting .profraw files to the same llvm-profdata merge call. More coverage, one profile.
-Db_pgo=generate binaries pay for counters on every branch and function call, and that’s not a rounding error.For a real release build, generate the profile on a plain optimized build first (faster iteration, and LTO doesn’t help you find hot paths, it just acts on them), then do the final build with both flags together:
meson configure build-pgo -Db_pgo=use -Db_lto=true -Db_lto_mode=thin
meson compile -C build-pgob_lto_mode=thin before you throw more RAM at it.default.profdata in the build directory means no -Db_pgo=use build.If you’re wondering whether any of this is worth the extra build complexity for your workload, that’s the actual subject of part 2: on a branch-light numerical loop, it bought me about 3%. Your mileage will vary with how branchy your hot path actually is.
Measuring PGO and LTO on a C++ SPH simulation with perf and flamegraphs, and why they barely helped.
Before touching a single line of my fluid simulation, I let the compiler do the work. Just changing flags got me from 1x to roughly 40x.
Multiboot2 drops you in 32-bit protected mode; Limine hands you a 64-bit environment. A single entry trampoline that handles both without forking the kernel.