--- title: Faster LLM inference startup: What you need to know description: Starting a new LLM inference replica is slow because the server must load weights, initialize GPU resources, compile kernels, and warm up before it can serve... canonical: https://daily.dev/posts/faster-llm-inference-startup-what-you-need-to-know-gtho9smmo twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Faster LLM inference startup: What you need to know | daily.dev og:description: Starting a new LLM inference replica is slow because the server must load weights, initialize GPU resources, compile kernels, and warm up before it can serve... og:url: https://daily.dev/posts/faster-llm-inference-startup-what-you-need-to-know-gtho9smmo og:image: https://api.daily.dev/og/posts/GtHO9sMMo.png og:image:alt: Faster LLM inference startup: What you need to know og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Faster LLM inference startup: What you need to know **[CNCF](https://daily.dev/sources/cncf)** · 11 min read · 1 upvotes · 0 comments ## Summary Starting a new LLM inference replica is slow because the server must load weights, initialize GPU resources, compile kernels, and warm up before it can serve requests. vLLM and SGLang are both building weight-cache daemons that hold processed weights in GPU memory and share them with new engines via CUDA IPC. vLLM's Fast Start daemon and preload command have merged, and vLLM also has an experimental CRIU-based snapshot feature limited to one GPU and tensor parallelism of one. SGLang has a similar weight-cache daemon and an open RFC for initialized-engine snapshots. Kubernetes has a pod-level checkpoint proposal (KEP-5823, with a v1.38 alpha target) and merged a CRI change for CheckpointPod and RestorePod on July 22, 2026, while GPU support (KEP-6274) remains a separate open proposal. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does vLLM's Fast Start weight daemon reduce engine startup time? It runs a persistent daemon that loads and processes model weights once, holding the resulting tensors in GPU memory, then exposes them to compatible engines through CUDA inter-process communication (IPC). In zero-copy mode, a new engine maps the daemon's existing allocations instead of reloading weights from disk. The daemon must stay alive while engines use its memory, and CUDA IPC handles are local to a node. _Teams tuning vLLM cold-start times can track Fast Start's progress and merge status on daily.dev._ ### What are the current limitations of vLLM's CRIU-based initialized-engine snapshot feature? The feature requires the same machine and a compatible environment, limited to one NVIDIA GPU and tensor parallelism of one. It uses CRIU to capture and restore Linux process state plus CUDA checkpoint support for GPU state, releasing weights and the KV cache before capture and reloading them during restoration, while still depending on the original model files. _Anyone evaluating snapshot-based cold-start fixes for vLLM can follow these compatibility constraints on daily.dev._ ### When did Kubernetes merge CRI support for checkpointing and restoring an entire Pod? The pull request adding CheckpointPod and RestorePod to the Container Runtime Interface merged on July 22, 2026. The interface defines consistent capture, resumption after checkpointing, and cleanup on restore failure, with restore preparing containers before execution starts. The Kubernetes orchestration and containerd implementation PRs remain open, and CRI-O's implementation is still a draft. _Platform engineers planning GPU workload checkpointing can follow Kubernetes Pod-level restore progress on daily.dev._ ## Similar posts on daily.dev - [Running LLM Inference on Kubernetes: What It Actually Takes](https://daily.dev/posts/running-llm-inference-on-kubernetes-what-it-actually-takes-opl6iozrg) · Fairwinds Blog · 0 upvotes · 0 comments - [NVIDIA Dynamo Snapshot: Fast Startup for Inference Workloads on Kubernetes](https://daily.dev/posts/nvidia-dynamo-snapshot-fast-startup-for-inference-workloads-on-kubernetes-eo8wvm8l4) · NVIDIA Developer · 1 upvotes · 0 comments - [How to achieve truly serverless GPUs](https://daily.dev/posts/how-to-achieve-truly-serverless-gpus-nggxyzsto) · Modal · 25 upvotes · 1 comments - [How NetEase Games cut LLM cold starts from 42 minutes to 30 seconds](https://daily.dev/posts/how-netease-games-cut-llm-cold-starts-from-42-minutes-to-30-seconds-gnzqqurwf) · The New Stack · 0 upvotes · 0 comments - [vLLM vs. SGLang: Choosing the right inference engine for your workload](https://daily.dev/posts/vllm-vs-sglang-choosing-the-right-inference-engine-for-your-workload-lirwkfyqx) · Red Hat Developer · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/faster-llm-inference-startup-what-you-need-to-know-gtho9smmo) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Faster LLM inference startup: What you need to know","url":"https://daily.dev/posts/faster-llm-inference-startup-what-you-need-to-know-gtho9smmo","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/faster-llm-inference-startup-what-you-need-to-know-gtho9smmo"},"datePublished":"2026-11-06T12:03:21.012Z","dateModified":"2026-11-06T12:03:49.407Z","description":"Starting a new LLM inference replica is slow because the server must load weights, initialize GPU resources, compile kernels, and warm up before it can serve...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fd5fe9c8ddd28513e6fb09c7e8275f44?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fd5fe9c8ddd28513e6fb09c7e8275f44?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"CNCF","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"CNCF","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/55536a03390741eebd96d2f9b74d2d8d","url":"https://daily.dev/sources/cncf"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/faster-llm-inference-startup-what-you-need-to-know-gtho9smmo","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,ai-inference,vllm","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"CNCF","item":"https://daily.dev/sources/cncf"},{"@type":"ListItem","position":3,"name":"Faster LLM inference startup: What you need to know"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/faster-llm-inference-startup-what-you-need-to-know-gtho9smmo#faq","mainEntity":[{"@type":"Question","name":"How does vLLM's Fast Start weight daemon reduce engine startup time?","acceptedAnswer":{"@type":"Answer","text":"It runs a persistent daemon that loads and processes model weights once, holding the resulting tensors in GPU memory, then exposes them to compatible engines through CUDA inter-process communication (IPC). In zero-copy mode, a new engine maps the daemon's existing allocations instead of reloading weights from disk. The daemon must stay alive while engines use its memory, and CUDA IPC handles are local to a node. Teams tuning vLLM cold-start times can track Fast Start's progress and merge status on daily.dev."}},{"@type":"Question","name":"What are the current limitations of vLLM's CRIU-based initialized-engine snapshot feature?","acceptedAnswer":{"@type":"Answer","text":"The feature requires the same machine and a compatible environment, limited to one NVIDIA GPU and tensor parallelism of one. It uses CRIU to capture and restore Linux process state plus CUDA checkpoint support for GPU state, releasing weights and the KV cache before capture and reloading them during restoration, while still depending on the original model files. Anyone evaluating snapshot-based cold-start fixes for vLLM can follow these compatibility constraints on daily.dev."}},{"@type":"Question","name":"When did Kubernetes merge CRI support for checkpointing and restoring an entire Pod?","acceptedAnswer":{"@type":"Answer","text":"The pull request adding CheckpointPod and RestorePod to the Container Runtime Interface merged on July 22, 2026. The interface defines consistent capture, resumption after checkpointing, and cleanup on restore failure, with restore preparing containers before execution starts. The Kubernetes orchestration and containerd implementation PRs remain open, and CRI-O's implementation is still a draft. Platform engineers planning GPU workload checkpointing can follow Kubernetes Pod-level restore progress on daily.dev."}}]} ```