--- title: When CPU Noise Slows Down GPU Inference: Measuring... description: An eBPF tracing tool called cuda_sched_trace was built to measure how Linux CPU scheduling and IRQ handling affect GPU LLM inference performance. Running Qwen3... canonical: https://daily.dev/posts/when-cpu-noise-slows-down-gpu-inference-measuring-scheduler-and-irq-impact-with-ebpf-szs0v9gse twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: When CPU Noise Slows Down GPU Inference: Measuring Scheduler and IRQ Impact with eBPF | daily.dev og:description: An eBPF tracing tool called cuda_sched_trace was built to measure how Linux CPU scheduling and IRQ handling affect GPU LLM inference performance. Running Qwen3... og:url: https://daily.dev/posts/when-cpu-noise-slows-down-gpu-inference-measuring-scheduler-and-irq-impact-with-ebpf-szs0v9gse og:image: https://api.daily.dev/og/posts/SzS0V9GSe.png og:image:alt: When CPU Noise Slows Down GPU Inference: Measuring Scheduler and IRQ Impact with eBPF og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # When CPU Noise Slows Down GPU Inference: Measuring Scheduler and IRQ Impact with eBPF **[eunomia eBPF Blog](https://daily.dev/sources/eunomia)** · 13 min read · 0 upvotes · 0 comments ## Summary An eBPF tracing tool called cuda_sched_trace was built to measure how Linux CPU scheduling and IRQ handling affect GPU LLM inference performance. Running Qwen3 0.6B inference under various noisy-neighbor conditions revealed that in a clean environment, scheduler impact is only 1.2% and IRQ impact is 0.03%. However, combined CPU, network, and disk interference causes 20.5% throughput degradation. Simple CPU pinning with taskset and nice reduces context switches by 96.3% and recovers most lost throughput. The study also compares findings with Meta's sched_ext work on distributed AI training, noting that local inference is dominated by CPU scheduling pressure rather than network IRQs, and provides practical recommendations for dedicated, shared, and Kubernetes environments. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much does CPU noisy-neighbor interference reduce GPU LLM inference throughput? Combined CPU, network, and disk interference reduces GPU LLM inference throughput by 20.5% compared to a clean baseline. CPU contention alone (stress-ng on all cores) causes an 8.8% drop with a 524x increase in context switches. Network load causes only 2.8% degradation, and disk I/O has negligible impact at -0.3%. The combined heavy-load case is worst due to cumulative effects. _Teams running GPU inference on shared hosts track interference patterns like these on daily.dev._ ### How effective is CPU pinning with taskset for reducing scheduler interference on GPU inference workloads? CPU pinning with taskset -c combined with nice -n -10 reduces context switches by 96.3% (from 11,932.8 to 445.2 per 1,000 kernel launches) and recovers 7.6% throughput under CPU-noisy conditions, bringing slowdown from 8.8% down to 1.9% versus a clean baseline. Full elimination is not possible because system daemons and IRQ affinity can still reach pinned cores. _Engineers optimizing GPU inference deployments on shared infrastructure follow mitigation techniques like this on daily.dev._ ### What is the IRQ overhead impact on local single-node LLM inference compared to distributed AI training? For local single-node LLM inference, IRQ overhead is negligible at 0.0276% of total runtime. This is because bursty kernel submission patterns (roughly 950 launches in under 100 microseconds per token) mean IRQs rarely land inside a burst. Distributed training with all-reduce network communication sees much higher IRQ impact, estimated at 5-20%, because NET_RX interrupts become a major bottleneck. _ML infrastructure engineers distinguishing single-node from distributed GPU workload behavior find relevant comparisons on daily.dev._ ## Similar posts on daily.dev - [Linux's sched\_ext Will Prioritize Idle SMT Siblings For Better Performance](https://daily.dev/posts/linux-s-sched-ext-will-prioritize-idle-smt-siblings-for-better-performance-tiery1ycf) · Phoronix · 0 upvotes · 0 comments - [A tale about fixing eBPF spinlock issues in the Linux kernel](https://daily.dev/posts/a-tale-about-fixing-ebpf-spinlock-issues-in-the-linux-kernel-rimv7upw4) · Hacker News · 1 upvotes · 0 comments - [Kernel Context Switch Costs And Spin Locks In Perf Profile: A Practical Study](https://daily.dev/posts/kernel-context-switch-costs-and-spin-locks-in-perf-profile-a-practical-study-joukt6jgh) · ITNEXT · 0 upvotes · 0 comments - [Intel's Cache Aware Scheduling Inches Closer To Being Merged For Linux](https://daily.dev/posts/intel-s-cache-aware-scheduling-inches-closer-to-being-merged-for-linux-otzzt47dg) · Phoronix · 1 upvotes · 0 comments --- Tags: [#linux](https://daily.dev/tags/linux), [#performance](https://daily.dev/tags/performance), [#cuda](https://daily.dev/tags/cuda) [View this post on daily.dev](https://daily.dev/posts/when-cpu-noise-slows-down-gpu-inference-measuring-scheduler-and-irq-impact-with-ebpf-szs0v9gse) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"When CPU Noise Slows Down GPU Inference: Measuring Scheduler and IRQ Impact with eBPF","url":"https://daily.dev/posts/when-cpu-noise-slows-down-gpu-inference-measuring-scheduler-and-irq-impact-with-ebpf-szs0v9gse","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/when-cpu-noise-slows-down-gpu-inference-measuring-scheduler-and-irq-impact-with-ebpf-szs0v9gse"},"datePublished":"2026-08-11T08:54:36.733Z","dateModified":"2026-08-11T09:32:45.600Z","description":"An eBPF tracing tool called cuda_sched_trace was built to measure how Linux CPU scheduling and IRQ handling affect GPU LLM inference performance. Running Qwen3...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/b59e04eefe0ea85df0b7e50151b0b12e?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/b59e04eefe0ea85df0b7e50151b0b12e?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"eunomia eBPF Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"eunomia eBPF Blog","logo":"https://media.daily.dev/image/upload/s--z-nTl7TK--/f_auto,q_auto/v1786438304/logos/eunomia?_a=BAMAMicg0","url":"https://daily.dev/sources/eunomia"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/when-cpu-noise-slows-down-gpu-inference-measuring-scheduler-and-irq-impact-with-ebpf-szs0v9gse","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"linux,performance,cuda","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"eunomia eBPF Blog","item":"https://daily.dev/sources/eunomia"},{"@type":"ListItem","position":3,"name":"When CPU Noise Slows Down GPU Inference: Measuring Scheduler and IRQ Impact with eBPF"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/when-cpu-noise-slows-down-gpu-inference-measuring-scheduler-and-irq-impact-with-ebpf-szs0v9gse#faq","mainEntity":[{"@type":"Question","name":"How much does CPU noisy-neighbor interference reduce GPU LLM inference throughput?","acceptedAnswer":{"@type":"Answer","text":"Combined CPU, network, and disk interference reduces GPU LLM inference throughput by 20.5% compared to a clean baseline. CPU contention alone (stress-ng on all cores) causes an 8.8% drop with a 524x increase in context switches. Network load causes only 2.8% degradation, and disk I/O has negligible impact at -0.3%. The combined heavy-load case is worst due to cumulative effects. Teams running GPU inference on shared hosts track interference patterns like these on daily.dev."}},{"@type":"Question","name":"How effective is CPU pinning with taskset for reducing scheduler interference on GPU inference workloads?","acceptedAnswer":{"@type":"Answer","text":"CPU pinning with taskset -c combined with nice -n -10 reduces context switches by 96.3% (from 11,932.8 to 445.2 per 1,000 kernel launches) and recovers 7.6% throughput under CPU-noisy conditions, bringing slowdown from 8.8% down to 1.9% versus a clean baseline. Full elimination is not possible because system daemons and IRQ affinity can still reach pinned cores. Engineers optimizing GPU inference deployments on shared infrastructure follow mitigation techniques like this on daily.dev."}},{"@type":"Question","name":"What is the IRQ overhead impact on local single-node LLM inference compared to distributed AI training?","acceptedAnswer":{"@type":"Answer","text":"For local single-node LLM inference, IRQ overhead is negligible at 0.0276% of total runtime. This is because bursty kernel submission patterns (roughly 950 launches in under 100 microseconds per token) mean IRQs rarely land inside a burst. Distributed training with all-reduce network communication sees much higher IRQ impact, estimated at 5-20%, because NET_RX interrupts become a major bottleneck. ML infrastructure engineers distinguishing single-node from distributed GPU workload behavior find relevant comparisons on daily.dev."}}]} ```