--- title: Grace Blackwell needs a new scheduler: 27% more usable... description: NVIDIA's NVL72 racks link all 72 GPUs on one NVLink fabric, but jobs only get the speedup if they stay rack-local. SkyPilot found that reserving 16-node slices... canonical: https://daily.dev/posts/grace-blackwell-needs-a-new-scheduler-27-more-usable-gpus-from-the-same-racks-fcqwfxzts twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Grace Blackwell needs a new scheduler: 27% more usable GPUs from the same racks | daily.dev og:description: NVIDIA's NVL72 racks link all 72 GPUs on one NVLink fabric, but jobs only get the speedup if they stay rack-local. SkyPilot found that reserving 16-node slices... og:url: https://daily.dev/posts/grace-blackwell-needs-a-new-scheduler-27-more-usable-gpus-from-the-same-racks-fcqwfxzts og:image: https://api.daily.dev/og/posts/FCQwfXztS.png og:image:alt: Grace Blackwell needs a new scheduler: 27% more usable GPUs from the same racks og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Grace Blackwell needs a new scheduler: 27% more usable GPUs from the same racks **[Hacker News](https://daily.dev/sources/hn)** · 11 min read · 0 upvotes · 0 comments ## Summary NVIDIA's NVL72 racks link all 72 GPUs on one NVLink fabric, but jobs only get the speedup if they stay rack-local. SkyPilot found that reserving 16-node slices with two spare nodes per rack gives 27% more usable capacity than filling racks with 18-node slices, based on measured 2% node unavailability. Their Topology-Aware Scheduler keeps multi-node jobs on a single rack, hot-swaps failed nodes using in-rack spares, and an Automatic Defragmentation feature preempts restartable jobs to free up whole racks for blocked workloads. NVIDIA's GR00T team saw rack-aware scheduling raise throughput by 14.4%, cutting iteration time from 1.59s to 1.39s, which on a 10,000-GPU cluster saves about $400K per training day. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### how many spare nodes per rack should I reserve on an NVIDIA GB200 NVL72 cluster for best usable capacity Sixteen-node slices with two spare nodes per rack deliver the most usable capacity on NVL72 clusters. At a measured 2% node unavailability rate, 16-node slices provide 27% more usable capacity than filling a full 18-node rack (2,037 vs. 1,602 usable nodes out of 2,304), and they beat 9-node slices by 4% because the first two node failures in a rack are fully absorbed by spares. _Teams sizing GPU clusters can track scheduling research like this on daily.dev._ ### why does running tensor parallelism or expert parallelism over InfiniBand instead of NVLink hurt training throughput Tensor parallelism requires two all-reduce synchronizations per transformer layer that sit on the critical path and cannot be fully overlapped with computation, so moving them off NVLink onto InfiniBand dropped training throughput by 43% in one measurement. Expert parallelism's all-to-all dispatch and combine steps can similarly consume up to 60% of training time when forced onto InfiniBand instead of NVLink. _Engineers debugging distributed training bottlenecks can follow interconnect benchmarks like this on daily.dev._ ### how much does rack-aware GPU scheduling improve training throughput compared to a naive scheduler NVIDIA's GR00T team measured a 14.4% throughput increase when scheduling 512 GB300 GPUs across 128 nodes with a rack-aware, slice-based policy placed on exactly 8 racks, versus a naive scheduler spreading nodes across an arbitrary number of racks. Iteration time dropped from 1.59 seconds to 1.39 seconds during pre-training of the GR00T N1.7 robotics model, saving an estimated $16.7K per training hour on a 10,000-GPU cluster. _Infra teams justifying scheduler investment can track benchmarks like this on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 29 comments across hackernews (as of 2026-10-29). **TL;DR:** Commenters largely dismiss the post as marketing fluff that states an already-obvious problem (topology-aware scheduling) without real technical detail, though a few authors engage in the thread to clarify specifics and promise follow-up posts. **Sentiment:** 10% positive · 25% mixed · 65% skeptical **The case for** - A few people found the problem framing (GPU failures in production on NVL72 systems) a useful starting point for discussion. - One commenter noted the topic is relevant to their own work building a similar scheduler, suggesting practical interest in the space. **The pushback** - Several say the post is light on technical detail about how the scheduler actually works, spending most of its length explaining basics. - Multiple commenters argue topology/gang-aware scheduling is old, well-known practice (since HTCondor and in most large GPU deployments), not a 'new' scheduler. - Some suspect the writing is AI-generated given the imbalance between lengthy background and thin technical substance. - It's called disingenuous/clickbaity and seen as shilling for the vendor. - A commenter questions whether reserving ~11% of expensive hardware as spares is actually a sensible tradeoff and suggests pooled spare capacity as an alternative. - Prior art from Kubernetes' Kueue topology-aware scheduling project is noted as unmentioned in the post. **By community** - hackernews (skeptical): Most engagement centers on calling the post marketing fluff that restates obvious, long-known scheduling concepts rather than revealing real technical depth, though the author(s) jump in to defend and promise more detailed follow-ups. **Hottest debate:** Whether this represents genuinely new scheduling work or just a repackaging of long-established topology/gang-aware scheduling techniques as marketing content. **Open questions** - How exactly does the topology-aware scheduler work internally beyond the two-paragraph description given? - Is reserving ~11% of nodes as spares the right tradeoff versus pooling spare capacity across racks/zones? - Will the scheduler (or related observability tooling) be open sourced? **Highlights** > This blog post spends a lot of time telling why gang scheduling is needed here (a fact obvious to anyone using the nvl72s), but doesn't actually go into detail about how their scheduler works. At work, we have built something very similar, a custom topology aware scheduler for workloads, and the most interesting things about such a project are entirely absent from this article, making it more of a fluff/marketing piece than an actually interesting technical blog in this area. Our next iteration of this scheduler aims to scale to O(millions) of GPUs, with a scheduling cycle time measured in the milliseconds. > — [jhgg on hackernews · 6 comments](https://news.ycombinator.com/item?id=50291965) > This reads a bit disingenuous and click-baity. There's no "new" scheduler. Gang scheduling and rack-aware scheduling have been around since the days of HTCondor, and are put in practice at almost every large GPU deployment I've seen. The net new information added by this blog is close to zero, and feels like shilling for SkyPilot. > — [novalite on hackernews · 1 comments](https://news.ycombinator.com/item?id=50293062) > This blog's goal is to introduce the problem around the new NVL72 system with GPU failures in production, which not all people outside of xAI are aware of. We have more blogs coming up for what we built to handle topology-aware scheduling, defragmentation and hot swapping during failures. The short version of the scheduling piece: jobs are decomposed into 16-node slices placed atomically on a single rack, with two spares per rack reserved for hot swaps. Seems very relevant to what you are building. Quoting what Ali said: "Quick PoC?" > — [zhwu on hackernews](https://news.ycombinator.com/item?id=50292775) > The danger of being excited about old techniques is giving the impression that you are years behind everyone else. A coworker once said of a particular breed of manager who is an emphatic late adopter: it’s as if every time he goes to the dentist he reads an old trade journal in the waiting room and comes back to work with a “new” idea that everyone else is already busy replacing. Are there any actual newer techniques? For instance, putting 11% of your hardware on reserve for hardware failures seems a bit generous to me for such an expensive cluster. Even if it’s down from some much larger number. One solution to this I’ve seen for handling cpu and memory variability with bin packing is to pool the spare capacity between tasks. For instance a data center with four zones might have 7 backup generators, instead of eight or twelve, counting on clusters of generator failures not to happen frequently. And of course there’s always running low priority tasks on the spare hardware and evicting them on a fault. However those low priority tasks tend to develop their own shadow priorities when they are in house. But that’s a management problem not a technical one. > — [hinkley on hackernews](https://news.ycombinator.com/item?id=50293464) > Yes, SkyPilot Platform built the scheduler on top of Kueue. There is some shortcoming of Kueue's TAS that you have to resolve, e.g., you cannot do TAS with a Elastic job, which makes the hot swapping not possible for large-scale jobs on NVL72 racks: https://kueue.sigs.k8s.io/docs/concepts/elastic_workload/#li... "No Topology-Aware Scheduling (TAS) support." There is another blog talking about this coming soon. > — [zhwu on hackernews](https://news.ycombinator.com/item?id=50294949) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50290912) · 102 points · 29 comments ## Similar posts on daily.dev - [Unlock Exascale Performance on NVIDIA GB200 NVL72 with Slurm Topology-Aware Job Scheduling](https://daily.dev/posts/unlock-exascale-performance-on-nvidia-gb200-nvl72-with-slurm-topology-aware-job-scheduling-tjbknhvnx) · NVIDIA Developer · 1 upvotes · 0 comments - [Impactful scheduling for GPU clusters](https://daily.dev/posts/impactful-scheduling-for-gpu-clusters-0wgrntsww) · Hugging Face · 6 upvotes · 1 comments - [Running AI Workloads on Rack-Scale Supercomputers: From Hardware to Topology-Aware Scheduling](https://daily.dev/posts/running-ai-workloads-on-rack-scale-supercomputers-from-hardware-to-topology-aware-scheduling-ralcmxfao) · NVIDIA Developer · 0 upvotes · 0 comments - [GPU Scheduling and Bin-Packing in Kubernetes: Pack More AI onto Every GPU](https://daily.dev/posts/gpu-scheduling-and-bin-packing-in-kubernetes-pack-more-ai-onto-every-gpu-6nuaafnff) · Cast AI · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning) [View this post on daily.dev](https://daily.dev/posts/grace-blackwell-needs-a-new-scheduler-27-more-usable-gpus-from-the-same-racks-fcqwfxzts) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Grace Blackwell needs a new scheduler: 27% more usable GPUs from the same racks","url":"https://daily.dev/posts/grace-blackwell-needs-a-new-scheduler-27-more-usable-gpus-from-the-same-racks-fcqwfxzts","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/grace-blackwell-needs-a-new-scheduler-27-more-usable-gpus-from-the-same-racks-fcqwfxzts"},"datePublished":"2026-10-29T17:24:11.546Z","dateModified":"2026-10-29T21:01:01.886Z","description":"NVIDIA's NVL72 racks link all 72 GPUs on one NVLink fabric, but jobs only get the speedup if they stay rack-local. SkyPilot found that reserving 16-node slices...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a09cf42424c8040c452bd09247351bde?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a09cf42424c8040c452bd09247351bde?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/grace-blackwell-needs-a-new-scheduler-27-more-usable-gpus-from-the-same-racks-fcqwfxzts","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Grace Blackwell needs a new scheduler: 27% more usable GPUs from the same racks"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/grace-blackwell-needs-a-new-scheduler-27-more-usable-gpus-from-the-same-racks-fcqwfxzts#faq","mainEntity":[{"@type":"Question","name":"how many spare nodes per rack should I reserve on an NVIDIA GB200 NVL72 cluster for best usable capacity","acceptedAnswer":{"@type":"Answer","text":"Sixteen-node slices with two spare nodes per rack deliver the most usable capacity on NVL72 clusters. At a measured 2% node unavailability rate, 16-node slices provide 27% more usable capacity than filling a full 18-node rack (2,037 vs. 1,602 usable nodes out of 2,304), and they beat 9-node slices by 4% because the first two node failures in a rack are fully absorbed by spares. Teams sizing GPU clusters can track scheduling research like this on daily.dev."}},{"@type":"Question","name":"why does running tensor parallelism or expert parallelism over InfiniBand instead of NVLink hurt training throughput","acceptedAnswer":{"@type":"Answer","text":"Tensor parallelism requires two all-reduce synchronizations per transformer layer that sit on the critical path and cannot be fully overlapped with computation, so moving them off NVLink onto InfiniBand dropped training throughput by 43% in one measurement. Expert parallelism's all-to-all dispatch and combine steps can similarly consume up to 60% of training time when forced onto InfiniBand instead of NVLink. Engineers debugging distributed training bottlenecks can follow interconnect benchmarks like this on daily.dev."}},{"@type":"Question","name":"how much does rack-aware GPU scheduling improve training throughput compared to a naive scheduler","acceptedAnswer":{"@type":"Answer","text":"NVIDIA's GR00T team measured a 14.4% throughput increase when scheduling 512 GB300 GPUs across 128 nodes with a rack-aware, slice-based policy placed on exactly 8 racks, versus a naive scheduler spreading nodes across an arbitrary number of racks. Iteration time dropped from 1.59 seconds to 1.39 seconds during pre-training of the GR00T N1.7 robotics model, saving an estimated $16.7K per training hour on a 10,000-GPU cluster. Infra teams justifying scheduler investment can track benchmarks like this on daily.dev."}}]} ```