--- title: How CUDA Graph Replay Cuts Inference Overhead | daily.dev description: CUDA Graph replay cuts GPU idle time during LLM decoding by removing repeated kernel-launch overhead. Normally the CPU submits each small GPU operation... canonical: https://daily.dev/posts/how-cuda-graph-replay-cuts-inference-overhead-dtcvtskee twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How CUDA Graph Replay Cuts Inference Overhead | daily.dev og:description: CUDA Graph replay cuts GPU idle time during LLM decoding by removing repeated kernel-launch overhead. Normally the CPU submits each small GPU operation... og:url: https://daily.dev/posts/how-cuda-graph-replay-cuts-inference-overhead-dtcvtskee og:image: https://api.daily.dev/og/posts/dtCVTSkee.png og:image:alt: How CUDA Graph Replay Cuts Inference Overhead og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How CUDA Graph Replay Cuts Inference Overhead **[Daily Dose of Data Science \| Avi Chawla \| Substack](https://daily.dev/sources/dailydoseofds)** · 7 min read · 1 upvotes · 0 comments ## Summary CUDA Graph replay cuts GPU idle time during LLM decoding by removing repeated kernel-launch overhead. Normally the CPU submits each small GPU operation (attention, matmul, normalization, sampling) one at a time, and if the GPU finishes before the next launch arrives, it sits idle. A CUDA Graph records a full sequence of operations and their dependencies once, during a capture step after warm-up. Replay then reruns that saved plan on new input data with a single graph-launch call instead of resubmitting every kernel. Production servers often capture multiple graphs for common batch sizes since a single graph can't cover every shape, trading extra memory and startup time for coverage. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is a CUDA graph and how does it reduce LLM inference latency? A CUDA graph is a saved execution plan that records a sequence of GPU operations, their arguments, and dependencies so they can be replayed with a single launch call instead of submitting each kernel individually. It is built in three steps: a warm-up pass, a capture pass that records the operations sent to a stream, and replay, where new input data is copied into the same buffers and the recorded graph reruns. This removes framework dispatch, argument setup, and driver call overhead from the steady-state decoding loop. _Developers tuning LLM serving latency can track techniques like this on daily.dev._ ### Why does the GPU sit idle during LLM token generation even though the model isn't finished computing? The GPU goes idle when it finishes a submitted kernel before the CPU prepares and sends the next one, since an LLM forward pass expands into many short kernels for attention, matrix multiplication, normalization, sampling, and memory movement, each requiring a separate launch through the framework, CUDA runtime, and driver. A profiler shows this as small gaps between GPU operations during decoding. _Engineers debugging GPU utilization gaps can follow inference optimization writeups on daily.dev._ ### Why do inference servers capture multiple CUDA graphs instead of just one? A single captured graph only works for the exact kernels, arguments, tensor shapes, control flow, and memory addresses used during capture, so production servers that handle varying batch sizes capture several graphs to cover common sizes. A batch either matches a captured graph, gets padded to a captured size, or falls back to normal execution; more captured graphs improve coverage but cost additional memory and startup time. _Teams sizing inference infrastructure can compare these tradeoffs on daily.dev._ ## Similar posts on daily.dev - [How I doubled my GPU efficiency without buying a single new card](https://daily.dev/posts/how-i-doubled-my-gpu-efficiency-without-buying-a-single-new-card-e57gspube) · InfoWorld · 0 upvotes · 0 comments - [Regular ML Inference vs. LLM Inference](https://daily.dev/posts/regular-ml-inference-vs-llm-inference-vwrarnvqn) · Daily Dose of Data Science \| Avi Chawla \| Substack · 6 upvotes · 0 comments - [Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo](https://daily.dev/posts/restore-llm-inference-capacity-in-seconds-with-shadow-engine-recovery-in-nvidia-dynamo-ucybjohkv) · NVIDIA Developer · 0 upvotes · 0 comments - [Prefill/Decode Disaggregation: Why Production LLM Inference Is Splitting Onto Separate Hardware](https://daily.dev/posts/prefill-decode-disaggregation-why-production-llm-inference-is-splitting-onto-separate-hardware-md2ahfjqb) · DigitalOcean Community · 0 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#rag](https://daily.dev/tags/rag), [#ai-inference](https://daily.dev/tags/ai-inference), [#cuda](https://daily.dev/tags/cuda) [View this post on daily.dev](https://daily.dev/posts/how-cuda-graph-replay-cuts-inference-overhead-dtcvtskee) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How CUDA Graph Replay Cuts Inference Overhead","url":"https://daily.dev/posts/how-cuda-graph-replay-cuts-inference-overhead-dtcvtskee","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-cuda-graph-replay-cuts-inference-overhead-dtcvtskee"},"datePublished":"2026-10-24T23:14:07.450Z","dateModified":"2026-10-24T23:14:40.089Z","description":"CUDA Graph replay cuts GPU idle time during LLM decoding by removing repeated kernel-launch overhead. Normally the CPU submits each small GPU operation...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7c26afe4e9ef8a772c2bbdff365b1688?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7c26afe4e9ef8a772c2bbdff365b1688?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Daily Dose of Data Science | Avi Chawla | Substack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Daily Dose of Data Science | Avi Chawla | Substack","logo":"https://media.daily.dev/image/upload/s--4IHQgTOw--/f_auto/v1710503712/logos/dailydoseofds","url":"https://daily.dev/sources/dailydoseofds"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-cuda-graph-replay-cuts-inference-overhead-dtcvtskee","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,rag,ai-inference,cuda","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Daily Dose of Data Science | Avi Chawla | Substack","item":"https://daily.dev/sources/dailydoseofds"},{"@type":"ListItem","position":3,"name":"How CUDA Graph Replay Cuts Inference Overhead"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-cuda-graph-replay-cuts-inference-overhead-dtcvtskee#faq","mainEntity":[{"@type":"Question","name":"What is a CUDA graph and how does it reduce LLM inference latency?","acceptedAnswer":{"@type":"Answer","text":"A CUDA graph is a saved execution plan that records a sequence of GPU operations, their arguments, and dependencies so they can be replayed with a single launch call instead of submitting each kernel individually. It is built in three steps: a warm-up pass, a capture pass that records the operations sent to a stream, and replay, where new input data is copied into the same buffers and the recorded graph reruns. This removes framework dispatch, argument setup, and driver call overhead from the steady-state decoding loop. Developers tuning LLM serving latency can track techniques like this on daily.dev."}},{"@type":"Question","name":"Why does the GPU sit idle during LLM token generation even though the model isn't finished computing?","acceptedAnswer":{"@type":"Answer","text":"The GPU goes idle when it finishes a submitted kernel before the CPU prepares and sends the next one, since an LLM forward pass expands into many short kernels for attention, matrix multiplication, normalization, sampling, and memory movement, each requiring a separate launch through the framework, CUDA runtime, and driver. A profiler shows this as small gaps between GPU operations during decoding. Engineers debugging GPU utilization gaps can follow inference optimization writeups on daily.dev."}},{"@type":"Question","name":"Why do inference servers capture multiple CUDA graphs instead of just one?","acceptedAnswer":{"@type":"Answer","text":"A single captured graph only works for the exact kernels, arguments, tensor shapes, control flow, and memory addresses used during capture, so production servers that handle varying batch sizes capture several graphs to cover common sizes. A batch either matches a captured graph, gets padded to a captured size, or falls back to normal execution; more captured graphs improve coverage but cost additional memory and startup time. Teams sizing inference infrastructure can compare these tradeoffs on daily.dev."}}]} ```