--- title: How DRA can improve the performance of distributed LLM... description: Dynamic resource allocation (DRA) in Kubernetes can fix a hidden performance trap in distributed LLM inference: pods getting a GPU and an RDMA NIC that are not... canonical: https://daily.dev/posts/how-dra-can-improve-the-performance-of-distributed-llm-inference-workloads-with-rdma-and-sr-iov-s7bruntwb twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How DRA can improve the performance of distributed LLM inference workloads with RDMA and SR-IOV | daily.dev og:description: Dynamic resource allocation (DRA) in Kubernetes can fix a hidden performance trap in distributed LLM inference: pods getting a GPU and an RDMA NIC that are not... og:url: https://daily.dev/posts/how-dra-can-improve-the-performance-of-distributed-llm-inference-workloads-with-rdma-and-sr-iov-s7bruntwb og:image: https://api.daily.dev/og/posts/s7bRUNTWB.png og:image:alt: How DRA can improve the performance of distributed LLM inference workloads with RDMA and SR-IOV og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How DRA can improve the performance of distributed LLM inference workloads with RDMA and SR-IOV **[Red Hat Developer](https://daily.dev/sources/rhdev)** · 9 min read · 0 upvotes · 0 comments ## Summary Dynamic resource allocation (DRA) in Kubernetes can fix a hidden performance trap in distributed LLM inference: pods getting a GPU and an RDMA NIC that are not on the same PCIe path. Traditional device plugins treat GPUs and SR-IOV virtual functions as opaque integer pools with no topology awareness, so a pod can pass scheduling but still route GPUDirect RDMA traffic across CPU sockets. Using dra-driver-sriov alongside the NVIDIA GPU DRA driver, a ResourceClaim can require a GPU and VF to share the same pcieRoot attribute. On an A100 platform with 200 Gbit/s InfiniBand, aligned GPU-NIC placement hit 194.6 Gbit/s versus 127.1 Gbit/s for same-NUMA misalignment and 110.0 Gbit/s cross-NUMA. In a real llm-d prefill/decode deployment on Red Hat OpenShift AI, aligned placement cut KV-cache transfer latency from 122.32 ms to 56.78 ms and more than doubled throughput to 20,709.6 MB/s. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much does GPU-NIC PCIe misalignment hurt RDMA bandwidth in Kubernetes with SR-IOV? Aligned GPU-NIC placement reached 194.6 Gbit/s on a 200 Gbit/s InfiniBand link, while same-NUMA but different-PCIe-root placement dropped to 127.1 Gbit/s (a 34.7% loss) and cross-NUMA placement fell to 110.0 Gbit/s (a 43.4% loss), measured with ib_write_bw using CUDA device memory, 8 MiB messages, and 5,000 iterations averaged over 10 runs. _Teams tuning RDMA-heavy GPU clusters can track topology-aware scheduling techniques like this on daily.dev._ ### How do I make Kubernetes schedule a GPU and an SR-IOV network VF on the same PCIe switch? Use dynamic resource allocation (DRA) with dra-driver-sriov and the NVIDIA GPU DRA driver, both of which publish a resource.kubernetes.io/pcieRoot attribute on their ResourceSlice devices. A ResourceClaim can apply a matchAttribute constraint requiring the allocated GPU and SR-IOV virtual function share the same pcieRoot value, so the scheduler only commits the pod when a topologically aligned pair exists on a node, leaving the claim pending otherwise. _Developers architecting GPU-NIC-aware Kubernetes scheduling can follow DRA developments like this on daily.dev._ ### What performance gain does aligning GPU and NIC placement give for disaggregated prefill/decode LLM serving? In a disaggregated prefill/decode deployment of Qwen3.8-27B using llm-d on Red Hat OpenShift AI with A100 GPUs, aligning the GPU and NIC under the same PCIe root cut mean KV-cache transfer latency from 122.32 ms to 56.78 ms and raised effective throughput from 9,612.6 MB/s to 20,709.6 MB/s, a 2.15x improvement, consistent across three test repetitions. _Anyone optimizing distributed LLM inference throughput can keep up with benchmarks like this via daily.dev._ ## Similar posts on daily.dev - [Kubernetes: Get the Most from Dynamic Resource Allocation](https://daily.dev/posts/kubernetes-get-the-most-from-dynamic-resource-allocation-hx4khv57l) · The New Stack · 2 upvotes · 0 comments - [Multitenant AI inference with dynamic resource allocation on OpenShift](https://daily.dev/posts/multitenant-ai-inference-with-dynamic-resource-allocation-on-openshift-krqyihtg7) · Red Hat Developer · 0 upvotes · 0 comments - [Understanding dynamic resource allocation in Kubernetes](https://daily.dev/posts/understanding-dynamic-resource-allocation-in-kubernetes-qr7yqzkhw) · CNCF · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#hardware](https://daily.dev/tags/hardware), [#ai-inference](https://daily.dev/tags/ai-inference) [View this post on daily.dev](https://daily.dev/posts/how-dra-can-improve-the-performance-of-distributed-llm-inference-workloads-with-rdma-and-sr-iov-s7bruntwb) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How DRA can improve the performance of distributed LLM inference workloads with RDMA and SR-IOV","url":"https://daily.dev/posts/how-dra-can-improve-the-performance-of-distributed-llm-inference-workloads-with-rdma-and-sr-iov-s7bruntwb","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-dra-can-improve-the-performance-of-distributed-llm-inference-workloads-with-rdma-and-sr-iov-s7bruntwb"},"datePublished":"2026-10-28T13:45:37.914Z","dateModified":"2026-10-28T15:03:30.022Z","description":"Dynamic resource allocation (DRA) in Kubernetes can fix a hidden performance trap in distributed LLM inference: pods getting a GPU and an RDMA NIC that are not...","image":"https://media.daily.dev/image/upload/s--VDukGCjf--/f_auto/v1722860399/public/Placeholder%2002","thumbnailUrl":"https://media.daily.dev/image/upload/s--VDukGCjf--/f_auto/v1722860399/public/Placeholder%2002","isAccessibleForFree":true,"articleSection":"Red Hat Developer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Red Hat Developer","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/2a8895f3216c4900bdc455585a4ad3e4","url":"https://daily.dev/sources/rhdev"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-dra-can-improve-the-performance-of-distributed-llm-inference-workloads-with-rdma-and-sr-iov-s7bruntwb","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,hardware,ai-inference","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Red Hat Developer","item":"https://daily.dev/sources/rhdev"},{"@type":"ListItem","position":3,"name":"How DRA can improve the performance of distributed LLM inference workloads with RDMA and SR-IOV"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-dra-can-improve-the-performance-of-distributed-llm-inference-workloads-with-rdma-and-sr-iov-s7bruntwb#faq","mainEntity":[{"@type":"Question","name":"How much does GPU-NIC PCIe misalignment hurt RDMA bandwidth in Kubernetes with SR-IOV?","acceptedAnswer":{"@type":"Answer","text":"Aligned GPU-NIC placement reached 194.6 Gbit/s on a 200 Gbit/s InfiniBand link, while same-NUMA but different-PCIe-root placement dropped to 127.1 Gbit/s (a 34.7% loss) and cross-NUMA placement fell to 110.0 Gbit/s (a 43.4% loss), measured with ib_write_bw using CUDA device memory, 8 MiB messages, and 5,000 iterations averaged over 10 runs. Teams tuning RDMA-heavy GPU clusters can track topology-aware scheduling techniques like this on daily.dev."}},{"@type":"Question","name":"How do I make Kubernetes schedule a GPU and an SR-IOV network VF on the same PCIe switch?","acceptedAnswer":{"@type":"Answer","text":"Use dynamic resource allocation (DRA) with dra-driver-sriov and the NVIDIA GPU DRA driver, both of which publish a resource.kubernetes.io/pcieRoot attribute on their ResourceSlice devices. A ResourceClaim can apply a matchAttribute constraint requiring the allocated GPU and SR-IOV virtual function share the same pcieRoot value, so the scheduler only commits the pod when a topologically aligned pair exists on a node, leaving the claim pending otherwise. Developers architecting GPU-NIC-aware Kubernetes scheduling can follow DRA developments like this on daily.dev."}},{"@type":"Question","name":"What performance gain does aligning GPU and NIC placement give for disaggregated prefill/decode LLM serving?","acceptedAnswer":{"@type":"Answer","text":"In a disaggregated prefill/decode deployment of Qwen3.8-27B using llm-d on Red Hat OpenShift AI with A100 GPUs, aligning the GPU and NIC under the same PCIe root cut mean KV-cache transfer latency from 122.32 ms to 56.78 ms and raised effective throughput from 9,612.6 MB/s to 20,709.6 MB/s, a 2.15x improvement, consistent across three test repetitions. Anyone optimizing distributed LLM inference throughput can keep up with benchmarks like this via daily.dev."}}]} ```