--- title: Cold Start Mechanics: How Warm Pools and Model Caching... description: Loading a 70B-parameter model onto a fresh GPU can take close to three minutes, and that wait, called a cold start, is why inference feels fast in a demo but... canonical: https://daily.dev/posts/cold-start-mechanics-how-warm-pools-and-model-caching-actually-work-ft0qvdunz twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Cold Start Mechanics: How Warm Pools and Model Caching Actually Work | daily.dev og:description: Loading a 70B-parameter model onto a fresh GPU can take close to three minutes, and that wait, called a cold start, is why inference feels fast in a demo but... og:url: https://daily.dev/posts/cold-start-mechanics-how-warm-pools-and-model-caching-actually-work-ft0qvdunz og:image: https://api.daily.dev/og/posts/ft0QvDunz.png og:image:alt: Cold Start Mechanics: How Warm Pools and Model Caching Actually Work og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Cold Start Mechanics: How Warm Pools and Model Caching Actually Work **[DigitalOcean Community](https://daily.dev/sources/do_community)** · 16 min read · 0 upvotes · 0 comments ## Summary Loading a 70B-parameter model onto a fresh GPU can take close to three minutes, and that wait, called a cold start, is why inference feels fast in a demo but uneven in production. A fully cold start can take 5 to 8 minutes once scheduling, image pull, weight download, GPU loading and engine initialization are all included. Caching weights on node-local storage cuts weight loading from roughly two minutes to under a minute. Adding warm node pools, engine compile caches and cached weights in CPU memory on warm nodes can shrink the total to 10 to 30 seconds, and a hot standby replica kept fully on GPU gives near zero start time at full GPU cost. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How long does it take to load a 70B parameter LLM onto a GPU from a cold start? A fully cold start for a 70B-parameter model takes roughly 5 to 8 minutes, covering node provisioning, container image pull, weight download, moving weights to GPU, and engine initialization. Weight movement alone is typically the largest slice, since the model's roughly 140GB of FP16 weights must be transferred from remote storage. _Teams sizing GPU autoscaling budgets can track inference infrastructure patterns like this on daily.dev._ ### How can I reduce vLLM startup time when autoscaling GPU replicas? Cache model weights locally on node NVMe or CPU memory instead of fetching from remote storage each time, which cuts weight loading from around two minutes to a few seconds. Also pre-pull the serving container image with a DaemonSet, reuse vLLM's torch.compile cache directory, and use vLLM's runai_streamer load format with concurrency around 16 to overlap disk reads and GPU copies. _Developers tuning vLLM deployments can follow infrastructure optimization techniques like these on daily.dev._ ### What is vLLM sleep mode and what are its requirements? vLLM sleep mode keeps a replica's weights in host memory and uses wake_up to restore them without a full reload, avoiding a cold restart. It requires enough host memory to hold the weights, and the HTTP sleep and wake endpoints need VLLM_SERVER_DEV_MODE=1 enabled; vLLM warns these development endpoints should not be exposed directly to users. _Engineers evaluating vLLM warm-state options can keep up with serving framework details via daily.dev._ ## Similar posts on daily.dev - [Cut GPU inference cold start from 8 minutes to less than a minute](https://daily.dev/posts/cut-gpu-inference-cold-start-from-8-minutes-to-less-than-a-minute-vhoj8trdz) · The New Stack · 0 upvotes · 0 comments - [Cold Start Latency on Serverless Inference: What Actually Happens in Those Few Seconds](https://daily.dev/posts/cold-start-latency-on-serverless-inference-what-actually-happens-in-those-few-seconds-0afuzoa3a) · DigitalOcean Community · 1 upvotes · 1 comments - [The Hidden Cost of Cold Starts in Serverless AI Workloads](https://daily.dev/posts/the-hidden-cost-of-cold-starts-in-serverless-ai-workloads-kptdj5xct) · DigitalOcean Community · 0 upvotes · 0 comments - [How to achieve truly serverless GPUs](https://daily.dev/posts/how-to-achieve-truly-serverless-gpus-nggxyzsto) · Modal · 25 upvotes · 1 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#gpu](https://daily.dev/tags/gpu), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/cold-start-mechanics-how-warm-pools-and-model-caching-actually-work-ft0qvdunz) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Cold Start Mechanics: How Warm Pools and Model Caching Actually Work","url":"https://daily.dev/posts/cold-start-mechanics-how-warm-pools-and-model-caching-actually-work-ft0qvdunz","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/cold-start-mechanics-how-warm-pools-and-model-caching-actually-work-ft0qvdunz"},"datePublished":"2026-10-26T22:07:40.189Z","dateModified":"2026-10-26T22:33:52.565Z","description":"Loading a 70B-parameter model onto a fresh GPU can take close to three minutes, and that wait, called a cold start, is why inference feels fast in a demo but...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ec6bbff1c611b994acc2033dd7d95398?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ec6bbff1c611b994acc2033dd7d95398?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"DigitalOcean Community","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"DigitalOcean Community","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/c1b9d07730e34ea388c39a498a753d6c","url":"https://daily.dev/sources/do_community"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/cold-start-mechanics-how-warm-pools-and-model-caching-actually-work-ft0qvdunz","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,gpu,ai-inference,vllm","timeRequired":"PT16M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DigitalOcean Community","item":"https://daily.dev/sources/do_community"},{"@type":"ListItem","position":3,"name":"Cold Start Mechanics: How Warm Pools and Model Caching Actually Work"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/cold-start-mechanics-how-warm-pools-and-model-caching-actually-work-ft0qvdunz#faq","mainEntity":[{"@type":"Question","name":"How long does it take to load a 70B parameter LLM onto a GPU from a cold start?","acceptedAnswer":{"@type":"Answer","text":"A fully cold start for a 70B-parameter model takes roughly 5 to 8 minutes, covering node provisioning, container image pull, weight download, moving weights to GPU, and engine initialization. Weight movement alone is typically the largest slice, since the model's roughly 140GB of FP16 weights must be transferred from remote storage. Teams sizing GPU autoscaling budgets can track inference infrastructure patterns like this on daily.dev."}},{"@type":"Question","name":"How can I reduce vLLM startup time when autoscaling GPU replicas?","acceptedAnswer":{"@type":"Answer","text":"Cache model weights locally on node NVMe or CPU memory instead of fetching from remote storage each time, which cuts weight loading from around two minutes to a few seconds. Also pre-pull the serving container image with a DaemonSet, reuse vLLM's torch.compile cache directory, and use vLLM's runai_streamer load format with concurrency around 16 to overlap disk reads and GPU copies. Developers tuning vLLM deployments can follow infrastructure optimization techniques like these on daily.dev."}},{"@type":"Question","name":"What is vLLM sleep mode and what are its requirements?","acceptedAnswer":{"@type":"Answer","text":"vLLM sleep mode keeps a replica's weights in host memory and uses wake_up to restore them without a full reload, avoiding a cold restart. It requires enough host memory to hold the weights, and the HTTP sleep and wake endpoints need VLLM_SERVER_DEV_MODE=1 enabled; vLLM warns these development endpoints should not be exposed directly to users. Engineers evaluating vLLM warm-state options can keep up with serving framework details via daily.dev."}}]} ```