--- title: Keeping 20,000 GPUs healthy | daily.dev description: Modal manages over 20,000 GPUs across AWS, GCP, Azure, and OCI, encountering significant reliability and performance differences between cloud providers. Their... canonical: https://daily.dev/posts/keeping-20-000-gpus-healthy-if9axbund twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Keeping 20,000 GPUs healthy | daily.dev og:description: Modal manages over 20,000 GPUs across AWS, GCP, Azure, and OCI, encountering significant reliability and performance differences between cloud providers. Their... og:url: https://daily.dev/posts/keeping-20-000-gpus-healthy-if9axbund og:image: https://api.daily.dev/og/posts/If9axbUND.png og:image:alt: Keeping 20,000 GPUs healthy og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Keeping 20,000 GPUs healthy **[Modal](https://daily.dev/sources/modal_labs)** · 10 min read · 40 upvotes · 3 comments ## Summary Modal manages over 20,000 GPUs across AWS, GCP, Azure, and OCI, encountering significant reliability and performance differences between cloud providers. Their GPU health system includes instance type benchmarking and selection, machine image preparation with automated testing, boot-time validation, and continuous passive monitoring (via DCGM and dmesg) plus weekly active healthchecks (DCGM diag, GPUBurn, NCCL tests). Key findings: Cloud providers vary dramatically in H100 performance (up to 50% differences), temperature management (some reaching 94°C), and ECC error rates. GPUs account for 58.7% of training failures in Meta's LLaMA 3 development, compared to just 0.5% for CPUs, highlighting the reliability gap. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community discussion Top comments from developers on daily.dev. **@shad001** · 9 upvotes > Allow them out in daylight for at least four hours a day, make sure that there's an ample supply of straw for their nests. GPUs like to make a fresh nest every other day, so this is paramount. > A diet of grains and root vegetables will ensure their fur stays thick and glossy, especially important for the cold, winter months. > GPUs prefer running water, so if you have access to a stream or river with shallow banks, so much the better. **@polliog** · 1 upvotes > If one of them is sick, I can take it ## Similar posts on daily.dev - [How we keep GPUs reliable across Databricks AI](https://daily.dev/posts/how-we-keep-gpus-reliable-across-databricks-ai-um3yzqfzg) · databricks · 0 upvotes · 0 comments - [Making GPU Clusters More Efficient with NVIDIA Data Center Monitoring](https://daily.dev/posts/making-gpu-clusters-more-efficient-with-nvidia-data-center-monitoring-hxuekxyge) · NVIDIA Developer · 1 upvotes · 0 comments - [GPU Cloud Pricing in 2026: What AI Compute Really Costs](https://daily.dev/posts/gpu-cloud-pricing-in-2026-what-ai-compute-really-costs-3qdtmbswu) · Cast AI · 0 upvotes · 0 comments - [How to achieve truly serverless GPUs](https://daily.dev/posts/how-to-achieve-truly-serverless-gpus-nggxyzsto) · Modal · 25 upvotes · 1 comments - [Beyond GPU Utilization: Building an End to End Observability Stack for LLM Inference on Kubernetes](https://daily.dev/posts/beyond-gpu-utilization-building-an-end-to-end-observability-stack-for-llm-inference-on-kubernetes-tjcterg7r) · Medium · 0 upvotes · 0 comments --- Tags: [#cloud](https://daily.dev/tags/cloud), [#nvidia](https://daily.dev/tags/nvidia), [#gpu](https://daily.dev/tags/gpu), [#monitoring](https://daily.dev/tags/monitoring) [View this post on daily.dev](https://daily.dev/posts/keeping-20-000-gpus-healthy-if9axbund) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Keeping 20,000 GPUs healthy","url":"https://daily.dev/posts/keeping-20-000-gpus-healthy-if9axbund","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/keeping-20-000-gpus-healthy-if9axbund"},"datePublished":"2026-01-09T19:12:29.684Z","dateModified":"2026-01-09T19:12:54.846Z","description":"Modal manages over 20,000 GPUs across AWS, GCP, Azure, and OCI, encountering significant reliability and performance differences between cloud providers. Their...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/6f9aa4b028c552acd1577445c74f53fe?_a=AQAEulh","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/6f9aa4b028c552acd1577445c74f53fe?_a=AQAEulh","isAccessibleForFree":true,"articleSection":"Modal","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Modal","logo":"https://media.daily.dev/image/upload/s--HfKZHzC1--/f_auto/v1750946306/logos/modal_labs","url":"https://daily.dev/sources/modal_labs"},"commentCount":3,"discussionUrl":"https://daily.dev/posts/keeping-20-000-gpus-healthy-if9axbund","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":40},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":3}],"keywords":"cloud,nvidia,gpu,monitoring","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Modal","item":"https://daily.dev/sources/modal_labs"},{"@type":"ListItem","position":3,"name":"Keeping 20,000 GPUs healthy"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/keeping-20-000-gpus-healthy-if9axbund","comment":[{"@type":"Comment","text":"Allow them out in daylight for at least four hours a day, make sure that there’s an ample supply of straw for their nests. GPUs like to make a fresh nest every other day, so this is paramount.\nA diet of grains and root vegetables will ensure their fur stays thick and glossy, especially important for the cold, winter months.\nGPUs prefer running water, so if you have access to a stream or river with shallow banks, so much the better.","datePublished":"2026-01-24T19:24:28.880Z","url":"https://daily.dev/posts/If9axbUND#c-BgKuHbc74","author":{"@type":"Person","name":"Shad","url":"https://daily.dev/shad001","image":"https://media.daily.dev/image/upload/s--O0TOmw4y--/f_auto/v1715772965/public/noProfile"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":9}},{"@type":"Comment","text":"If one of them is sick, I can take it","datePublished":"2026-02-15T12:16:51.778Z","url":"https://daily.dev/posts/If9axbUND#c-AqANRu8xm","author":{"@type":"Person","name":"Polliog","url":"https://daily.dev/polliog","image":"https://avatars.githubusercontent.com/u/40077351?v=4"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}]} ```