Intel's iaprof is an open-source GPU profiler that generates AI Flame Graphs by combining three data streams: eBPF kernel tracing of GPU driver calls, Intel Observability Architecture (OA) hardware sampling of Execution Unit stalls, and a GPU debug API for shader binary retrieval. The result is a unified visualization connecting Python/C++ application code through runtime libraries and kernel drivers down to specific GPU shader instructions, annotated with stall type breakdowns (memory, control flow, synchronization). The tool uses a multi-threaded architecture with a central GPU kernel store, two-level locking, deferred attribution for out-of-order samples, and outputs folded stack text compatible with Brendan Gregg's FlameGraph tooling. Overhead ranges from 5–10% on the Xe driver to 15–30% on i915. Supported hardware includes Intel Data Center GPU Max (Ponte Vecchio), Arc B-series (Battlemage), and other Xe2-based GPUs, requiring Linux 5.8+ with BTF support.
Table of contents
The Visibility Gap in GPU ComputingHow iaprof Bridges the GapIntel GPU Architecture BackgroundThe Architecture: Inside iaprofSystem OverviewCore ComponentsExecution FlowKey Design DecisionsPerformance CharacteristicsPlatform SupportFuture EnhancementsReferencesFlame Graph Generation PipelineWhat This Means for eBPF-Based GPU ObservabilityQuestions this post answers
How does Intel iaprof correlate CPU call stacks with GPU shader instructions?
iaprof uses three concurrent data streams: eBPF fexit probes on i915/Xe driver functions capture the full CPU call stack at every GPU kernel launch; Intel's Observability Architecture hardware samples EU stall states at specific GPU instruction pointers; and the GPU debug API provides shader binaries for disassembly. A central GPU kernel store merges these streams, mapping each stalled GPU instruction back through the shader to the originating CPU stack frame. Engineers profiling GPU-accelerated ML workloads track tools like iaprof on daily.dev.
What is the profiling overhead of Intel iaprof on Xe vs i915 GPU drivers?
On the Xe driver, total overhead is 5–10%: BPF collection adds 3–5%, EU stall sampling 2–5%, and the debug interface 1–3%. On the older i915 driver, overhead is higher at 15–30% total, with BPF collection alone costing 10–20% due to more complex batch buffer parsing. EU stall hardware sampling contributes under 5% in both cases. Developers choosing between profiling strategies for Intel GPU workloads find trade-off breakdowns like this on daily.dev.
What Linux kernel version does Intel iaprof require and why?
iaprof requires Linux 5.8 or later for eBPF CO-RE (Compile Once, Run Everywhere) support, plus BTF type information at /sys/kernel/btf/vmlinux. For Xe GPU debug support, mainline Linux 6.2+ is needed, with full support in 6.6+. i915 debug support requires an Intel backport kernel with PRELIM_DRM_I915_DEBUG patches, which is not in mainline. Developers upgrading kernel stacks for GPU observability keep up with Linux and driver requirements on daily.dev.