--- title: GPU virtualization at scale: AMD MI300X SR-IOV on Red... description: A validation of AMD Instinct MI300X GPU virtualization using hardware SR-IOV on an 8-GPU system running Red Hat OpenStack Services on OpenShift is described,... canonical: https://daily.dev/posts/gpu-virtualization-at-scale-amd-mi300x-sr-iov-on-red-hat-openstack-services-on-openshift-lxqujj8cf twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: GPU virtualization at scale: AMD MI300X SR-IOV on Red Hat OpenStack Services on OpenShift | daily.dev og:description: A validation of AMD Instinct MI300X GPU virtualization using hardware SR-IOV on an 8-GPU system running Red Hat OpenStack Services on OpenShift is described,... og:url: https://daily.dev/posts/gpu-virtualization-at-scale-amd-mi300x-sr-iov-on-red-hat-openstack-services-on-openshift-lxqujj8cf og:image: https://api.daily.dev/og/posts/LXqUJJ8CF.png og:image:alt: GPU virtualization at scale: AMD MI300X SR-IOV on Red Hat OpenStack Services on OpenShift og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # GPU virtualization at scale: AMD MI300X SR-IOV on Red Hat OpenStack Services on OpenShift **[Red Hat Developer](https://daily.dev/sources/rhdev)** · 16 min read · 0 upvotes · 0 comments ## Summary A validation of AMD Instinct MI300X GPU virtualization using hardware SR-IOV on an 8-GPU system running Red Hat OpenStack Services on OpenShift is described, covering architecture, configuration steps, and performance results. GIM partitions each MI300X GPU into 8 virtual functions (VFs) in CPX mode, producing 64 schedulable GPU VFs that Nova and Placement inventory and assign via VFIO PCI passthrough. Testing exercised configurations from 64 VMs with 1 VF each up to 32 VFs per VM, with the 64-VF single-VM case still under investigation due to virtual-IOMMU issues. BabelStream benchmarks showed 96-99% of bare-metal memory bandwidth, while CoralGemm testing showed physical GPU placement still affects throughput. IOMMU isolation blocks direct GPU peer-to-peer DMA between VFs, making SR-IOV best suited for independent workloads like inference rather than tightly coupled multi-GPU jobs. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much memory bandwidth do you lose when using SR-IOV virtual functions on an AMD MI300X GPU compared to bare metal? SR-IOV virtual functions on an AMD Instinct MI300X deliver approximately 96-99% of bare-metal GPU memory bandwidth for primary BabelStream streaming operations, tested with FP64 data on an 805 MB dataset over 100 iterations. The Dot operation showed different scaling behavior and was interpreted separately from this primary result. This was measured using AMD GIM driver 8.6.0 in CPX mode with 8 VFs per GPU on ROCm 7.1.1. _Developers benchmarking GPU virtualization overhead can track real-world validation data like this on daily.dev._ ### Can GPUs communicate directly with each other when using SR-IOV virtual functions on AMD MI300X? No, direct GPU-to-GPU peer-to-peer DMA communication between isolated virtual functions is blocked by IOMMU isolation in SR-IOV configurations tested on the AMD Instinct MI300X. Bare-metal XGMI bandwidth measured 503-518 GB/s unidirectional and 840-926 GB/s bidirectional, but CPU-to-GPU transfers in VFs are limited to around 42 GB/s. This makes SR-IOV suited to independent workloads like inference and RAG rather than tightly coupled multi-GPU applications. _Architects weighing SR-IOV against full GPU passthrough can compare these tradeoffs on daily.dev._ ### How many virtual GPU functions can you create from 8 AMD MI300X GPUs using SR-IOV in CPX mode? Eight physical AMD Instinct MI300X GPUs in CPX mode can expose 64 total SR-IOV virtual functions, with each GPU partitioned into 8 VFs using the AMD GIM driver (version 8.6.0). Each CPX VF exposes approximately 24 GB of VRAM. A maximum-density test of 64 virtual machines, each assigned one VF, passed successfully, while configurations up to 32 VFs on a single VM were also validated. _Teams planning multi-tenant GPU density on OpenStack can follow validated configurations like this on daily.dev._ ## Similar posts on daily.dev - [Virtualizing NVidia HGX B200 GPUs with Open Source](https://daily.dev/posts/virtualizing-nvidia-hgx-b200-gpus-with-open-source-qaefumqmz) · Hacker News · 0 upvotes · 0 comments - [Intel Posts Patches For New VFIO Xe PCI Linux Driver](https://daily.dev/posts/intel-posts-patches-for-new-vfio-xe-pci-linux-driver-pc9aqkhvu) · Phoronix · 0 upvotes · 0 comments - [AMD Working On A New Backend For Improving ROCm Compute In QEMU/VMs](https://daily.dev/posts/amd-working-on-a-new-backend-for-improving-rocm-compute-in-qemu-vms-px3wwgxqw) · Phoronix · 0 upvotes · 0 comments - [Network performance in distributed training](https://daily.dev/posts/network-performance-in-distributed-training-f2cvwbf5i) · Red Hat Developer · 1 upvotes · 0 comments --- Tags: [#rocm](https://daily.dev/tags/rocm) [View this post on daily.dev](https://daily.dev/posts/gpu-virtualization-at-scale-amd-mi300x-sr-iov-on-red-hat-openstack-services-on-openshift-lxqujj8cf) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"GPU virtualization at scale: AMD MI300X SR-IOV on Red Hat OpenStack Services on OpenShift","url":"https://daily.dev/posts/gpu-virtualization-at-scale-amd-mi300x-sr-iov-on-red-hat-openstack-services-on-openshift-lxqujj8cf","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/gpu-virtualization-at-scale-amd-mi300x-sr-iov-on-red-hat-openstack-services-on-openshift-lxqujj8cf"},"datePublished":"2026-10-01T13:21:12.469Z","dateModified":"2026-10-01T13:21:37.399Z","description":"A validation of AMD Instinct MI300X GPU virtualization using hardware SR-IOV on an 8-GPU system running Red Hat OpenStack Services on OpenShift is described,...","image":"https://media.daily.dev/image/upload/s--P4t4XyoV--/f_auto/v1722860399/public/Placeholder%2001","thumbnailUrl":"https://media.daily.dev/image/upload/s--P4t4XyoV--/f_auto/v1722860399/public/Placeholder%2001","isAccessibleForFree":true,"articleSection":"Red Hat Developer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Red Hat Developer","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/2a8895f3216c4900bdc455585a4ad3e4","url":"https://daily.dev/sources/rhdev"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/gpu-virtualization-at-scale-amd-mi300x-sr-iov-on-red-hat-openstack-services-on-openshift-lxqujj8cf","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"rocm","timeRequired":"PT16M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Red Hat Developer","item":"https://daily.dev/sources/rhdev"},{"@type":"ListItem","position":3,"name":"GPU virtualization at scale: AMD MI300X SR-IOV on Red Hat OpenStack Services on OpenShift"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/gpu-virtualization-at-scale-amd-mi300x-sr-iov-on-red-hat-openstack-services-on-openshift-lxqujj8cf#faq","mainEntity":[{"@type":"Question","name":"How much memory bandwidth do you lose when using SR-IOV virtual functions on an AMD MI300X GPU compared to bare metal?","acceptedAnswer":{"@type":"Answer","text":"SR-IOV virtual functions on an AMD Instinct MI300X deliver approximately 96-99% of bare-metal GPU memory bandwidth for primary BabelStream streaming operations, tested with FP64 data on an 805 MB dataset over 100 iterations. The Dot operation showed different scaling behavior and was interpreted separately from this primary result. This was measured using AMD GIM driver 8.6.0 in CPX mode with 8 VFs per GPU on ROCm 7.1.1. Developers benchmarking GPU virtualization overhead can track real-world validation data like this on daily.dev."}},{"@type":"Question","name":"Can GPUs communicate directly with each other when using SR-IOV virtual functions on AMD MI300X?","acceptedAnswer":{"@type":"Answer","text":"No, direct GPU-to-GPU peer-to-peer DMA communication between isolated virtual functions is blocked by IOMMU isolation in SR-IOV configurations tested on the AMD Instinct MI300X. Bare-metal XGMI bandwidth measured 503-518 GB/s unidirectional and 840-926 GB/s bidirectional, but CPU-to-GPU transfers in VFs are limited to around 42 GB/s. This makes SR-IOV suited to independent workloads like inference and RAG rather than tightly coupled multi-GPU applications. Architects weighing SR-IOV against full GPU passthrough can compare these tradeoffs on daily.dev."}},{"@type":"Question","name":"How many virtual GPU functions can you create from 8 AMD MI300X GPUs using SR-IOV in CPX mode?","acceptedAnswer":{"@type":"Answer","text":"Eight physical AMD Instinct MI300X GPUs in CPX mode can expose 64 total SR-IOV virtual functions, with each GPU partitioned into 8 VFs using the AMD GIM driver (version 8.6.0). Each CPX VF exposes approximately 24 GB of VRAM. A maximum-density test of 64 virtual machines, each assigned one VF, passed successfully, while configurations up to 32 VFs on a single VM were also validated. Teams planning multi-tenant GPU density on OpenStack can follow validated configurations like this on daily.dev."}}]} ```