NVIDIA introduces shadow engine recovery, a preview feature in NVIDIA Dynamo that dramatically speeds up recovery from LLM inference engine failures. Instead of a cold restart that reloads weights, compiles kernels, and recaptures CUDA graphs, a fully initialized shadow engine sits idle on the same GPUs sharing weights via a GPU Memory Service (GMS). When the active engine fails, the shadow takes over in seconds instead of minutes. Benchmarks on a two-worker GLM-5.2 deployment on NVIDIA B200 GPUs show recovery time dropping from 283 seconds to 7.3 seconds, cutting TTFT from 23,815ms to 1,311ms and improving decode rate from 12 to 46 tok/s/user. The feature requires Kubernetes 1.34+ with Dynamic Resource Allocation and currently supports vLLM primarily, with KV cache materialization still deferred during promotion.
Table of contents
Why LLM inference recovery is slow: two core problemsHow shadow engine recovery worksRecovery in depthBenchmark results: shadow engine recovery vs. cold restart on GLM-5.2Current scope and next stepsQuestions this post answers
What is shadow engine recovery in NVIDIA Dynamo and how fast is it compared to a cold restart?
Shadow engine recovery is a preview feature in NVIDIA Dynamo that keeps a fully initialized standby engine idle on the same GPUs as the active engine, sharing weights via a GPU Memory Service so it can take over within seconds after a failure. In a benchmark with a two-worker GLM-5.2 deployment on NVIDIA B200 GPUs, recovery time dropped from 283 seconds with a cold restart to 7.3 seconds, roughly 39 times faster. Teams building resilient LLM serving pipelines can track failover techniques like this one on daily.dev.
What are the requirements to use shadow engine recovery in NVIDIA Dynamo?
Shadow engine recovery requires Dynamic Resource Allocation (DRA) on Kubernetes, meaning the cluster needs Kubernetes 1.34 or newer with DRA enabled and the NVIDIA GPU DRA driver installed. vLLM is currently the primary supported backend, and the feature does not yet cover hardware, node, or multi-node failures, which still rely on standard rescheduling. Engineers evaluating Kubernetes GPU infrastructure upgrades can follow rollout details for tools like this on daily.dev.
How does the GPU Memory Service (GMS) let two inference engines share model weights without duplicating them in GPU memory?
GMS is a per-GPU sidecar process that owns physical GPU memory independently of any engine process, using the CUDA Virtual Memory Management API so physical allocations and virtual addresses have independent lifetimes. Engines import handles and map the same physical pages into their own CUDA contexts, so a second engine incurs zero marginal weight cost and weights survive even if an engine process crashes. Developers optimizing GPU memory usage in inference systems can follow architectural deep dives like this on daily.dev.