--- title: Kernel Context Switch Costs And Spin Locks In Perf... description: A deep-dive investigation explains why perf profiling showed heavy CPU costs attributed to finish_task_switch and _raw_spin_unlock_irqrestore in a 40-process... canonical: https://daily.dev/posts/kernel-context-switch-costs-and-spin-locks-in-perf-profile-a-practical-study-joukt6jgh twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Kernel Context Switch Costs And Spin Locks In Perf Profile: A Practical Study | daily.dev og:description: A deep-dive investigation explains why perf profiling showed heavy CPU costs attributed to finish_task_switch and _raw_spin_unlock_irqrestore in a 40-process... og:url: https://daily.dev/posts/kernel-context-switch-costs-and-spin-locks-in-perf-profile-a-practical-study-joukt6jgh og:image: https://api.daily.dev/og/posts/joUkt6JgH.png og:image:alt: Kernel Context Switch Costs And Spin Locks In Perf Profile: A Practical Study og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Kernel Context Switch Costs And Spin Locks In Perf Profile: A Practical Study **[ITNEXT](https://daily.dev/sources/itnext)** · 12 min read · 0 upvotes · 0 comments ## Summary A deep-dive investigation explains why perf profiling showed heavy CPU costs attributed to finish_task_switch and _raw_spin_unlock_irqrestore in a 40-process benchmark on a 12-core EC2 VM using SysV semaphores for locking. The apparent spinlock cost turns out to be a measurement artifact: perf's timer interrupts are blocked while interrupts are disabled inside the kernel scheduler, so time is misattributed to the unlock function that finally re-enables interrupts. The real cost comes from heavy work inside schedule() and try_to_wake_up(), including runqueue spinlock contention and smp_cond_load_acquire busy-spinning while waiting for the sleeping task to finish its context switch. The author walks through Linux scheduler source (try_to_wake_up, ttwu_queue_wakelist, rq_lock) to explain the mechanism and suggests CPU affinity and reduced preemption as mitigations. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does perf show heavy CPU usage in _raw_spin_unlock_irqrestore when profiling semaphore-heavy workloads on Linux? It is a measurement artifact, not an actual slow spinlock. Perf relies on timer-based interrupts to sample CPU usage, but raw_spin_lock_irqsave disables interrupts while holding a lock, so perf's interrupt cannot fire until raw_spin_unlock_irqrestore calls local_irq_restore. All the CPU time consumed between disabling and restoring interrupts gets wrongly billed to the unlock function itself. _When profiling scheduler hotspots on daily.dev readers can dig into how perf interrupt sampling can mislead spinlock analysis._ ### How many times does try_to_wake_up() acquire the runqueue lock in the Linux scheduler? It grabs rq_lock twice in the normal case, via ttwu_queue() to enqueue the wakeup task on a new CPU and psi_ttwu_dequeue() for pressure stall metrics, and up to three times in the worst case when ttwu_runnable() also takes the old runqueue's lock. Combined with potential busy-spinning in smp_cond_load_acquire, a single wakeup can be blocked by up to four context switches. _Anyone tuning scheduler contention can compare notes on runqueue locking behavior via daily.dev._ ### What causes try_to_wake_up to burn CPU cycles while waiting for a sleeping process to context switch out? The smp_cond_load_acquire call busy-spins until the sleeping task's p->on_cpu flag is cleared to 0, which only happens once that task finishes context switching out inside schedule(). Under heavy CPU-bound contention with more processes than cores, this spin can last as long as the peer's full context switch, and the ttwu_queue_wakelist shortcut that would normally avoid this spin gets skipped whenever the target CPU's runqueue already has other runnable tasks (nr_running > 0). _Developers chasing wakeup latency under contention can track scheduler internals discussions like this on daily.dev._ ## Similar posts on daily.dev - [A tale about fixing eBPF spinlock issues in the Linux kernel](https://daily.dev/posts/a-tale-about-fixing-ebpf-spinlock-issues-in-the-linux-kernel-rimv7upw4) · Hacker News · 1 upvotes · 0 comments - [New Patches Work To Optimize Code Generation For Linux Context Switching](https://daily.dev/posts/new-patches-work-to-optimize-code-generation-for-linux-context-switching-a8xejsqse) · Phoronix · 0 upvotes · 0 comments - [Netflix Uncovers Kernel-Level Bottlenecks While Scaling Containers on Modern CPUs](https://daily.dev/posts/netflix-uncovers-kernel-level-bottlenecks-while-scaling-containers-on-modern-cpus-bfs33g6uj) · InfoQ · 3 upvotes · 0 comments - [Spinning around: Please don't\!](https://daily.dev/posts/spinning-around-please-don-t--cekmueark) · Hacker News · 0 upvotes · 0 comments - [Context Switches](https://daily.dev/posts/context-switches-3g5ft7ffm) · matklad · 1 upvotes · 0 comments --- Tags: [#linux](https://daily.dev/tags/linux) [View this post on daily.dev](https://daily.dev/posts/kernel-context-switch-costs-and-spin-locks-in-perf-profile-a-practical-study-joukt6jgh) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Kernel Context Switch Costs And Spin Locks In Perf Profile: A Practical Study","url":"https://daily.dev/posts/kernel-context-switch-costs-and-spin-locks-in-perf-profile-a-practical-study-joukt6jgh","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/kernel-context-switch-costs-and-spin-locks-in-perf-profile-a-practical-study-joukt6jgh"},"datePublished":"2026-08-14T09:21:02.647Z","dateModified":"2026-09-13T18:44:06.231Z","description":"A deep-dive investigation explains why perf profiling showed heavy CPU costs attributed to finish_task_switch and _raw_spin_unlock_irqrestore in a 40-process...","image":"https://media.daily.dev/image/upload/s--HRgLpUt6--/f_auto/v1722860399/public/Placeholder%2003","thumbnailUrl":"https://media.daily.dev/image/upload/s--HRgLpUt6--/f_auto/v1722860399/public/Placeholder%2003","isAccessibleForFree":true,"articleSection":"ITNEXT","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ITNEXT","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/itnext2","url":"https://daily.dev/sources/itnext"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/kernel-context-switch-costs-and-spin-locks-in-perf-profile-a-practical-study-joukt6jgh","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"linux","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ITNEXT","item":"https://daily.dev/sources/itnext"},{"@type":"ListItem","position":3,"name":"Kernel Context Switch Costs And Spin Locks In Perf Profile: A Practical Study"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/kernel-context-switch-costs-and-spin-locks-in-perf-profile-a-practical-study-joukt6jgh#faq","mainEntity":[{"@type":"Question","name":"Why does perf show heavy CPU usage in _raw_spin_unlock_irqrestore when profiling semaphore-heavy workloads on Linux?","acceptedAnswer":{"@type":"Answer","text":"It is a measurement artifact, not an actual slow spinlock. Perf relies on timer-based interrupts to sample CPU usage, but raw_spin_lock_irqsave disables interrupts while holding a lock, so perf's interrupt cannot fire until raw_spin_unlock_irqrestore calls local_irq_restore. All the CPU time consumed between disabling and restoring interrupts gets wrongly billed to the unlock function itself. When profiling scheduler hotspots on daily.dev readers can dig into how perf interrupt sampling can mislead spinlock analysis."}},{"@type":"Question","name":"How many times does try_to_wake_up() acquire the runqueue lock in the Linux scheduler?","acceptedAnswer":{"@type":"Answer","text":"It grabs rq_lock twice in the normal case, via ttwu_queue() to enqueue the wakeup task on a new CPU and psi_ttwu_dequeue() for pressure stall metrics, and up to three times in the worst case when ttwu_runnable() also takes the old runqueue's lock. Combined with potential busy-spinning in smp_cond_load_acquire, a single wakeup can be blocked by up to four context switches. Anyone tuning scheduler contention can compare notes on runqueue locking behavior via daily.dev."}},{"@type":"Question","name":"What causes try_to_wake_up to burn CPU cycles while waiting for a sleeping process to context switch out?","acceptedAnswer":{"@type":"Answer","text":"The smp_cond_load_acquire call busy-spins until the sleeping task's p->on_cpu flag is cleared to 0, which only happens once that task finishes context switching out inside schedule(). Under heavy CPU-bound contention with more processes than cores, this spin can last as long as the peer's full context switch, and the ttwu_queue_wakelist shortcut that would normally avoid this spin gets skipped whenever the target CPU's runqueue already has other runnable tasks (nr_running > 0). Developers chasing wakeup latency under contention can track scheduler internals discussions like this on daily.dev."}}]} ```