--- title: Ceph.io — For whom the door-bell tolls | daily.dev description: A deep engineering deep-dive describes building rados-nkv, a Ceph/RADOS-backed NVMe key-value target, then extending it to support fully GPU-initiated IO using... canonical: https://daily.dev/posts/ceph-io-for-whom-the-door-bell-tolls-erbgn5qif twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Ceph.io — For whom the door-bell tolls | daily.dev og:description: A deep engineering deep-dive describes building rados-nkv, a Ceph/RADOS-backed NVMe key-value target, then extending it to support fully GPU-initiated IO using... og:url: https://daily.dev/posts/ceph-io-for-whom-the-door-bell-tolls-erbgn5qif og:image: https://api.daily.dev/og/posts/eRbGn5qIF.png og:image:alt: Ceph.io — For whom the door-bell tolls og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Ceph.io — For whom the door-bell tolls **[Ceph](https://daily.dev/sources/ceph)** · 29 min read · 0 upvotes · 0 comments ## Summary A deep engineering deep-dive describes building rados-nkv, a Ceph/RADOS-backed NVMe key-value target, then extending it to support fully GPU-initiated IO using AMD's ROCm XIO on a Strix Halo Mini PC. The author walks through prototyping with vfio-user and a QEMU pcie-mmio-bridge, hitting and debugging three distinct bugs (a guest-physical address collision with the VGA framebuffer, an NVMe queue-size mismatch, and a torn completion-queue read), and ultimately demonstrates a GPU issuing a key-value Retrieve command whose value is DMA'd directly from a RADOS object into GPU VRAM with no CPU in the data path, achieving ~140µs latency. The piece closes with throughput math on batched submission queues for LLM KV-cache prefixes and a roadmap for computational storage via allowlisted RADOS classes. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What causes an NVMe CQ poll timeout with a 0x1016 HSA_STATUS_ERROR_EXCEPTION when doing GPU-initiated NVMe IO with ROCm XIO? It signals the GPU's completion-queue poll loop hit its built-in trap after giving up waiting for a completion, but the root cause found in one debugging session was not GPU-related at all: a shadow ring placed at guest-physical address 0x80000000 collided with QEMU's standard VGA framebuffer BAR, so doorbell writes vanished into video memory instead of reaching the NVMe controller. Moving the shadow ring above 4GB fixed it. _Anyone debugging GPU passthrough or P2P DMA quirks can find deep dives like this through daily.dev._ ### Why does SPDK reject NVMe queue creation with error 'invalid I/O queue size 1024'? SPDK's advertised controller capability (CAP register) reported MQES (Maximum Queue Entries Supported) of only 255, meaning a max queue size of 256 entries, while the client hard-coded 1024-entry queues, causing SPDK to reject both CREATE_IO_CQ and CREATE_IO_SQ. Raising the transport's queue depth with 'nvmf_create_transport -t VFIOUSER -q 1024 -m 16' bumped MQES to 1023 and resolved it. _Developers tuning SPDK NVMe-oF transports for large queue depths can track these low-level fixes on daily.dev._ ### Why would a key-value NVMe interface be better than a block interface for LLM KV cache storage? A key-value interface lets the hash of a token sequence serve directly as the storage key, so two GPUs computing the same prefix land on the same object with zero coordination, whereas a block interface requires a centralized lookup table mapping cache-block hashes to (device, offset, length) tuples. A prototype demonstrated a GPU issuing an NVMe KV Retrieve that P2P-DMAs a value straight from a RADOS object into GPU VRAM at about 140µs per IO, matching block-read latency once bridge polling overhead is removed. _Teams designing LLM inference storage layers can follow content-addressed caching approaches like this on daily.dev._ ## Similar posts on daily.dev - [GPU virtualization at scale: AMD MI300X SR-IOV on Red Hat OpenStack Services on OpenShift](https://daily.dev/posts/gpu-virtualization-at-scale-amd-mi300x-sr-iov-on-red-hat-openstack-services-on-openshift-lxqujj8cf) · Red Hat Developer · 0 upvotes · 0 comments - [AMD Working On A New Backend For Improving ROCm Compute In QEMU/VMs](https://daily.dev/posts/amd-working-on-a-new-backend-for-improving-rocm-compute-in-qemu-vms-px3wwgxqw) · Phoronix · 0 upvotes · 0 comments - [How DOCA GPUNetIO Unifies GPU-Initiated Networking Across the NVIDIA Software Stack](https://daily.dev/posts/how-doca-gpunetio-unifies-gpu-initiated-networking-across-the-nvidia-software-stack-hmk8elgzk) · NVIDIA Developer · 0 upvotes · 0 comments - [MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet](https://daily.dev/posts/metaroce-a-new-rdma-transport-built-for-ai-scale-ethernet-4rzmhaxu4) · Facebook Engineering · 0 upvotes · 0 comments - [Igniting the GPU: From Kernel Plumbing to 3D Rendering on RISC-V](https://daily.dev/posts/igniting-the-gpu-from-kernel-plumbing-to-3d-rendering-on-risc-v-n3d4nu797) · Hacker News · 1 upvotes · 0 comments --- [View this post on daily.dev](https://daily.dev/posts/ceph-io-for-whom-the-door-bell-tolls-erbgn5qif) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Ceph.io — For whom the door-bell tolls","url":"https://daily.dev/posts/ceph-io-for-whom-the-door-bell-tolls-erbgn5qif","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/ceph-io-for-whom-the-door-bell-tolls-erbgn5qif"},"datePublished":"2026-08-23T12:17:37.645Z","dateModified":"2026-08-23T12:26:29.043Z","description":"A deep engineering deep-dive describes building rados-nkv, a Ceph/RADOS-backed NVMe key-value target, then extending it to support fully GPU-initiated IO using...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/33c943fddc446912efca0602d972db76?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/33c943fddc446912efca0602d972db76?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Ceph","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Ceph","logo":"https://media.daily.dev/image/upload/s--nHzZM4Ap--/f_auto,q_auto/v1787487446/logos/ceph?_a=BAMAMicg0","url":"https://daily.dev/sources/ceph"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/ceph-io-for-whom-the-door-bell-tolls-erbgn5qif","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"","timeRequired":"PT29M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Ceph","item":"https://daily.dev/sources/ceph"},{"@type":"ListItem","position":3,"name":"Ceph.io — For whom the door-bell tolls"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/ceph-io-for-whom-the-door-bell-tolls-erbgn5qif#faq","mainEntity":[{"@type":"Question","name":"What causes an NVMe CQ poll timeout with a 0x1016 HSA_STATUS_ERROR_EXCEPTION when doing GPU-initiated NVMe IO with ROCm XIO?","acceptedAnswer":{"@type":"Answer","text":"It signals the GPU's completion-queue poll loop hit its built-in trap after giving up waiting for a completion, but the root cause found in one debugging session was not GPU-related at all: a shadow ring placed at guest-physical address 0x80000000 collided with QEMU's standard VGA framebuffer BAR, so doorbell writes vanished into video memory instead of reaching the NVMe controller. Moving the shadow ring above 4GB fixed it. Anyone debugging GPU passthrough or P2P DMA quirks can find deep dives like this through daily.dev."}},{"@type":"Question","name":"Why does SPDK reject NVMe queue creation with error 'invalid I/O queue size 1024'?","acceptedAnswer":{"@type":"Answer","text":"SPDK's advertised controller capability (CAP register) reported MQES (Maximum Queue Entries Supported) of only 255, meaning a max queue size of 256 entries, while the client hard-coded 1024-entry queues, causing SPDK to reject both CREATE_IO_CQ and CREATE_IO_SQ. Raising the transport's queue depth with 'nvmf_create_transport -t VFIOUSER -q 1024 -m 16' bumped MQES to 1023 and resolved it. Developers tuning SPDK NVMe-oF transports for large queue depths can track these low-level fixes on daily.dev."}},{"@type":"Question","name":"Why would a key-value NVMe interface be better than a block interface for LLM KV cache storage?","acceptedAnswer":{"@type":"Answer","text":"A key-value interface lets the hash of a token sequence serve directly as the storage key, so two GPUs computing the same prefix land on the same object with zero coordination, whereas a block interface requires a centralized lookup table mapping cache-block hashes to (device, offset, length) tuples. A prototype demonstrated a GPU issuing an NVMe KV Retrieve that P2P-DMAs a value straight from a RADOS object into GPU VRAM at about 140µs per IO, matching block-read latency once bridge polling overhead is removed. Teams designing LLM inference storage layers can follow content-addressed caching approaches like this on daily.dev."}}]} ```