LLM inference workloads expose significant OS-level bottlenecks that limit performance beyond raw compute. Key challenges include memory management (fragmentation, page faults, GPU-CPU swapping), CPU scheduling jitter (default Linux CFS can cause worst-case delays of 11ms vs. ~14µs with core isolation), I/O latency from model loading and page faults, and real-time constraint violations from OS noise. Solutions explored include eBPF-based custom scheduling (e.g., Google's ghOSt framework, openEuler's programmable scheduler), smarter memory prefetching via FetchBPF, transparent huge pages, NUMA-aware allocation (5–10% gains from local NUMA access), kernel bypass techniques (DPDK, SPDK, io_uring), and emerging CXL tiered memory. Security considerations cover multi-tenant isolation, seccomp filtering, least-privilege principles, side-channel risks from SMT, and confidential computing via TEEs. Practical guidance covers profiling baselines with perf/iostat, incremental changes, eBPF vs. kernel module trade-offs, and evaluating tail latency under realistic workloads.
Table of contents
OS-Level Bottlenecks in LLM InferenceCustom Kernel Extensions and eBPF for AI WorkloadsSystem Calls, Page Faults, and Kernel–User Space InteractionsSecurity and Isolation in AI Inference WorkloadsOS Customization for AI: Best Practices and Emerging ResearchPractical Considerations for Implementing OS-Level OptimizationsQuestions this post answers
How much latency jitter does Linux's default scheduler add to LLM inference and how can I reduce it?
Without tuning, Linux's default CFS scheduler can delay one in 100 events by over 2 milliseconds, with worst-case delays reaching 11ms. Pinning inference threads to dedicated CPU cores and isolating them from normal scheduling can reduce worst-case latency to around 14 microseconds — roughly a 1000x improvement. Techniques include using the isolcpus kernel parameter, taskset or cset shield, and real-time scheduling policies. Engineers tuning LLM serving latency track scheduler isolation techniques like these on daily.dev.
How can eBPF be used to optimize memory prefetching for LLM inference workloads?
eBPF programs can hook into kernel page fault events to monitor fault rates on model memory and trigger proactive prefetching before pages are needed. Research prototypes like FetchBPF implement customizable prefetching policies in Linux via eBPF. For LLM inference, an eBPF program can recognize sequential layer scanning and pre-fault the next layer from disk or host memory, avoiding stalls during inference. Developers building LLM infrastructure find eBPF and kernel optimization research on daily.dev.
What is the performance benefit of NUMA-aware memory allocation for deep learning inference on multi-socket CPUs?
Keeping memory on the same NUMA node as the CPU core performing inference can improve performance by 5–10% compared to cross-node memory access. This is because cross-NUMA memory access incurs additional latency over the interconnect. Custom kernel modules or Linux's existing NUMA policies can enforce local allocation for model weights and intermediate tensors. Teams deploying inference on multi-socket servers follow NUMA optimization findings on daily.dev.