--- title: One kernel feature, 93% system throughput gone: A Red... description: A kernel scheduler regression in RHEL 10.2 caused up to 93% throughput loss on CockroachDB workloads running on multi-NUMA AWS instances. The regression is... canonical: https://daily.dev/posts/one-kernel-feature-93-system-throughput-gone-a-red-hat-enterprise-linux-10-2-kernel-regression-an-zefh5aaqc twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: One kernel feature, 93% system throughput gone: A Red Hat Enterprise Linux 10.2 kernel regression and how to mitigate it | daily.dev og:description: A kernel scheduler regression in RHEL 10.2 caused up to 93% throughput loss on CockroachDB workloads running on multi-NUMA AWS instances. The regression is... og:url: https://daily.dev/posts/one-kernel-feature-93-system-throughput-gone-a-red-hat-enterprise-linux-10-2-kernel-regression-an-zefh5aaqc og:image: https://api.daily.dev/og/posts/ZEFH5aAQC.png og:image:alt: One kernel feature, 93% system throughput gone: A Red Hat Enterprise Linux 10.2 kernel regression and how to mitigate it og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # One kernel feature, 93% system throughput gone: A Red Hat Enterprise Linux 10.2 kernel regression and how to mitigate it **[Red Hat Developer](https://daily.dev/sources/rhdev)** · 9 min read · 14 upvotes · 1 comments ## Summary A kernel scheduler regression in RHEL 10.2 caused up to 93% throughput loss on CockroachDB workloads running on multi-NUMA AWS instances. The regression is bimodal — systems either run at full speed or enter a severe CPU work starvation state where CPUs sit 85-95% idle despite runnable threads waiting. The culprit is the NI_RANDOM scheduler feature, introduced to reduce overhead from failed newidle balance attempts, which inadvertently starves runnable threads on multi-NUMA hardware. The fix requires no reboot: disabling NI_RANDOM via a single sysfs write restores full throughput. A tmpfiles drop-in can persist the setting across reboots. The investigation involved vmstat analysis, thread stack traces, schedstat data, and targeted kernel reverts using Red Hat's kbisect-dbs tool to narrow 18,848 commits down to the responsible scheduler changes. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I fix the RHEL 10.2 kernel scheduler regression that causes massive throughput loss on multi-NUMA systems? Disable the NI_RANDOM scheduler feature at runtime by running echo NO_NI_RANDOM > /sys/kernel/debug/sched/features, which requires no reboot. To persist the fix across reboots, create a tmpfiles drop-in at /etc/tmpfiles.d/sched_features.conf setting NO_NI_RANDOM. This resolves a bug where randomized newidle balancing starves runnable threads on multi-NUMA hardware, restoring full throughput until an upstream fix lands in RHEL. _Engineers hunting scheduler regressions like this one can track kernel fixes and workarounds on daily.dev._ ### What causes the bimodal throughput collapse on RHEL 10.2 multi-NUMA instances running CockroachDB benchmarks? Randomized newidle balancing, introduced by the NI_RANDOM scheduler feature, creates a feedback loop that starves runnable threads of CPU time on multi-NUMA hardware. The system flips between two stable states: full throughput or up to 93% degraded, with CPUs sitting 85-95% idle despite 128+ runnable processes, because the scheduler stops distributing wake-ups to threads waiting on Go runtime futex locks. _Teams benchmarking database workloads on new kernel releases can follow regression writeups like this via daily.dev._ ### Which AWS instance types are most affected by the RHEL 10.2 newidle scheduler regression? Large multi-NUMA instances suffer the worst degradation: m6a.24xlarge (96 vCPUs, 2 NUMA nodes, AMD EPYC 7R13) dropped 68%, m5.24xlarge (96 vCPUs, Xeon 8175M) dropped 65%, m7a.16xlarge (64 vCPUs, AMD EPYC 9R14) dropped 31%, and m7i.16xlarge (64 vCPUs, Xeon 8488C) dropped 15%. Small single-NUMA instances remained largely unaffected across 29 instance types tested. _Engineers choosing cloud instance types for database workloads can weigh regression reports like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@trevorsuna** · 3 upvotes > A 93% regression with mostly idle CPUs is a great reminder that CPU utilization alone can mislead. Run-queue and NUMA-aware metrics would make this failure easier to spot, and the no-reboot mitigation is especially valuable for busy database nodes. ## Similar posts on daily.dev - [Linux 7.0 Slab Fix On The Way For A "Severe Performance Regression"](https://daily.dev/posts/linux-7-0-slab-fix-on-the-way-for-a-severe-performance-regression--nj6t21vbt) · Phoronix · 0 upvotes · 0 comments - [Netflix Uncovers Kernel-Level Bottlenecks While Scaling Containers on Modern CPUs](https://daily.dev/posts/netflix-uncovers-kernel-level-bottlenecks-while-scaling-containers-on-modern-cpus-bfs33g6uj) · InfoQ · 3 upvotes · 0 comments --- Tags: [#linux](https://daily.dev/tags/linux), [#cockroachdb](https://daily.dev/tags/cockroachdb) [View this post on daily.dev](https://daily.dev/posts/one-kernel-feature-93-system-throughput-gone-a-red-hat-enterprise-linux-10-2-kernel-regression-an-zefh5aaqc) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"One kernel feature, 93% system throughput gone: A Red Hat Enterprise Linux 10.2 kernel regression and how to mitigate it","url":"https://daily.dev/posts/one-kernel-feature-93-system-throughput-gone-a-red-hat-enterprise-linux-10-2-kernel-regression-an-zefh5aaqc","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/one-kernel-feature-93-system-throughput-gone-a-red-hat-enterprise-linux-10-2-kernel-regression-an-zefh5aaqc"},"datePublished":"2026-08-04T03:35:15.483Z","dateModified":"2026-09-13T21:44:53.289Z","description":"A kernel scheduler regression in RHEL 10.2 caused up to 93% throughput loss on CockroachDB workloads running on multi-NUMA AWS instances. The regression is...","image":"https://media.daily.dev/image/upload/s--foaA6JGU--/f_auto/v1722860399/public/Placeholder%2004","thumbnailUrl":"https://media.daily.dev/image/upload/s--foaA6JGU--/f_auto/v1722860399/public/Placeholder%2004","isAccessibleForFree":true,"articleSection":"Red Hat Developer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Red Hat Developer","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/2a8895f3216c4900bdc455585a4ad3e4","url":"https://daily.dev/sources/rhdev"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/one-kernel-feature-93-system-throughput-gone-a-red-hat-enterprise-linux-10-2-kernel-regression-an-zefh5aaqc","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":14},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"linux,cockroachdb","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Red Hat Developer","item":"https://daily.dev/sources/rhdev"},{"@type":"ListItem","position":3,"name":"One kernel feature, 93% system throughput gone: A Red Hat Enterprise Linux 10.2 kernel regression and how to mitigate it"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/one-kernel-feature-93-system-throughput-gone-a-red-hat-enterprise-linux-10-2-kernel-regression-an-zefh5aaqc","comment":[{"@type":"Comment","text":"A 93% regression with mostly idle CPUs is a great reminder that CPU utilization alone can mislead. Run-queue and NUMA-aware metrics would make this failure easier to spot, and the no-reboot mitigation is especially valuable for busy database nodes.","datePublished":"2026-08-04T05:12:48.558Z","url":"https://daily.dev/posts/ZEFH5aAQC#c-bfjp8Q3IG","author":{"@type":"Person","name":"Trevor Suna","url":"https://daily.dev/trevorsuna","image":"https://media.daily.dev/image/upload/s--dZ7gXxpp--/f_auto/v1784081551/avatars/avatar_EMoP47rpuw8DNjhp6R1b6?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/one-kernel-feature-93-system-throughput-gone-a-red-hat-enterprise-linux-10-2-kernel-regression-an-zefh5aaqc#faq","mainEntity":[{"@type":"Question","name":"How do I fix the RHEL 10.2 kernel scheduler regression that causes massive throughput loss on multi-NUMA systems?","acceptedAnswer":{"@type":"Answer","text":"Disable the NI_RANDOM scheduler feature at runtime by running echo NO_NI_RANDOM > /sys/kernel/debug/sched/features, which requires no reboot. To persist the fix across reboots, create a tmpfiles drop-in at /etc/tmpfiles.d/sched_features.conf setting NO_NI_RANDOM. This resolves a bug where randomized newidle balancing starves runnable threads on multi-NUMA hardware, restoring full throughput until an upstream fix lands in RHEL. Engineers hunting scheduler regressions like this one can track kernel fixes and workarounds on daily.dev."}},{"@type":"Question","name":"What causes the bimodal throughput collapse on RHEL 10.2 multi-NUMA instances running CockroachDB benchmarks?","acceptedAnswer":{"@type":"Answer","text":"Randomized newidle balancing, introduced by the NI_RANDOM scheduler feature, creates a feedback loop that starves runnable threads of CPU time on multi-NUMA hardware. The system flips between two stable states: full throughput or up to 93% degraded, with CPUs sitting 85-95% idle despite 128+ runnable processes, because the scheduler stops distributing wake-ups to threads waiting on Go runtime futex locks. Teams benchmarking database workloads on new kernel releases can follow regression writeups like this via daily.dev."}},{"@type":"Question","name":"Which AWS instance types are most affected by the RHEL 10.2 newidle scheduler regression?","acceptedAnswer":{"@type":"Answer","text":"Large multi-NUMA instances suffer the worst degradation: m6a.24xlarge (96 vCPUs, 2 NUMA nodes, AMD EPYC 7R13) dropped 68%, m5.24xlarge (96 vCPUs, Xeon 8175M) dropped 65%, m7a.16xlarge (64 vCPUs, AMD EPYC 9R14) dropped 31%, and m7i.16xlarge (64 vCPUs, Xeon 8488C) dropped 15%. Small single-NUMA instances remained largely unaffected across 29 instance types tested. Engineers choosing cloud instance types for database workloads can weigh regression reports like this on daily.dev."}}]} ```