--- title: Infra behind Krea 2: How to train and serve at scale —... description: A Krea.ai infrastructure engineer describes how the Krea 2 image generation model was pre-trained from scratch across thousands of Infiniband-connected GPUs,... canonical: https://daily.dev/posts/infra-behind-krea-2-how-to-train-and-serve-at-scale-gabriel-jorge-menezes-krea-ai-ht7lrd8oq twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai | daily.dev og:description: A Krea.ai infrastructure engineer describes how the Krea 2 image generation model was pre-trained from scratch across thousands of Infiniband-connected GPUs,... og:url: https://daily.dev/posts/infra-behind-krea-2-how-to-train-and-serve-at-scale-gabriel-jorge-menezes-krea-ai-ht7lrd8oq og:image: https://api.daily.dev/og/posts/ht7lrD8oq.png og:image:alt: Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai **[AI Engineer](https://daily.dev/sources/aidotengineer)** · 16 min read · 0 upvotes · 0 comments ## Summary A Krea.ai infrastructure engineer describes how the Krea 2 image generation model was pre-trained from scratch across thousands of Infiniband-connected GPUs, and how the company serves it in production. Key lessons include distrusting GPU utilization metrics in favor of tensor core utilization, building custom InfiniBand and NVLink error monitoring since NVIDIA's DCGM doesn't export InfiniBand metrics, aggressively checkpointing to survive frequent training crashes, and using a paid high-throughput filesystem after Ceph proved unreliable. The talk also covers a Kubernetes-based scheduling system using Kueue for gang scheduling and Virtual Kubelet to dynamically flip GPU capacity between production inference and training workloads using taints, tolerations, and a descheduler, maximizing GPU utilization without manual intervention. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why is GPU utilization a misleading metric during large-scale model training? GPU utilization only measures whether the GPU is doing any work at all, not how efficiently it's being used, so it can show 100% even when the hardware is far from fully utilized. Tensor core utilization is a better proxy because it reflects how much of the tensor cores are actually being exercised, which rises as image resolution scales up during pre-training. _Anyone debugging GPU cluster performance can find more real-world infra lessons like this on daily.dev._ ### Why doesn't NVIDIA's DCGM export InfiniBand metrics by default for GPU clusters? DCGM exports some NVLink metrics but no InfiniBand metrics out of the box, so teams running large multi-node training clusters have to build custom collection for InfiniBand throughput, wait times, packet counts, and error rates themselves. These metrics turned out critical because most training failures traced back to cross-node communication problems, not the GPUs themselves. _Engineers building GPU monitoring pipelines can track similar infra deep dives through daily.dev._ ### How can Kubernetes dynamically shift GPU capacity between production inference and model training? Kubernetes taints and tolerations combined with Virtual Kubelet can automatically detect when a cluster's GPUs are needed for training and taint the nodes so inference pods get evicted, then use a descheduler to gradually migrate inference pods back once training finishes and GPUs free up. A gradual migration avoids a hard 'no-execute' taint that would kick everything out at once and cause production downtime. _Teams balancing GPU costs between training and serving can find similar Kubernetes patterns on daily.dev._ ## Similar posts on daily.dev - [Building Blocks for Foundation Model Training and Inference on AWS](https://daily.dev/posts/building-blocks-for-foundation-model-training-and-inference-on-aws-e3dmqot6y) · Hugging Face · 1 upvotes · 0 comments - [Building a Kubernetes-native pattern for AI infrastructure at scale](https://daily.dev/posts/building-a-kubernetes-native-pattern-for-ai-infrastructure-at-scale-20cemnrwo) · The New Stack · 0 upvotes · 0 comments - [Building a reliable cloud native foundation for distributed AI training](https://daily.dev/posts/building-a-reliable-cloud-native-foundation-for-distributed-ai-training-pwun78uwc) · CNCF · 1 upvotes · 0 comments - [Kubernetes Did Not Miss the AI Wave. It Absorbed It](https://daily.dev/posts/kubernetes-did-not-miss-the-ai-wave-it-absorbed-it-d4xqnmisz) · Cloud Native Now · 5 upvotes · 4 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#gpu](https://daily.dev/tags/gpu), [#diffusion-models](https://daily.dev/tags/diffusion-models) [View this post on daily.dev](https://daily.dev/posts/infra-behind-krea-2-how-to-train-and-serve-at-scale-gabriel-jorge-menezes-krea-ai-ht7lrd8oq) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai","url":"https://daily.dev/posts/infra-behind-krea-2-how-to-train-and-serve-at-scale-gabriel-jorge-menezes-krea-ai-ht7lrd8oq","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/infra-behind-krea-2-how-to-train-and-serve-at-scale-gabriel-jorge-menezes-krea-ai-ht7lrd8oq"},"datePublished":"2026-08-18T17:20:33.018Z","dateModified":"2026-08-18T17:21:00.281Z","description":"A Krea.ai infrastructure engineer describes how the Krea 2 image generation model was pre-trained from scratch across thousands of Infiniband-connected GPUs,...","image":"https://i.ytimg.com/vi/byn9PURoBNY/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/byn9PURoBNY/sddefault.jpg","isAccessibleForFree":true,"articleSection":"AI Engineer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AI Engineer","logo":"https://media.daily.dev/image/upload/s--u5PucxNT--/f_auto/v1724338940/logos/aidotengineer","url":"https://daily.dev/sources/aidotengineer"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/infra-behind-krea-2-how-to-train-and-serve-at-scale-gabriel-jorge-menezes-krea-ai-ht7lrd8oq","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,gpu,diffusion-models","timeRequired":"PT16M","video":{"@type":"VideoObject","name":"Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai","description":"A Krea.ai infrastructure engineer describes how the Krea 2 image generation model was pre-trained from scratch across thousands of Infiniband-connected GPUs,...","thumbnailUrl":"https://i.ytimg.com/vi/byn9PURoBNY/sddefault.jpg","uploadDate":"2026-08-18T17:20:33.018Z","duration":"PT16M","url":"https://api.daily.dev/r/ht7lrD8oq","embedUrl":"https://www.youtube.com/embed/byn9PURoBNY"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI Engineer","item":"https://daily.dev/sources/aidotengineer"},{"@type":"ListItem","position":3,"name":"Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/infra-behind-krea-2-how-to-train-and-serve-at-scale-gabriel-jorge-menezes-krea-ai-ht7lrd8oq#faq","mainEntity":[{"@type":"Question","name":"Why is GPU utilization a misleading metric during large-scale model training?","acceptedAnswer":{"@type":"Answer","text":"GPU utilization only measures whether the GPU is doing any work at all, not how efficiently it's being used, so it can show 100% even when the hardware is far from fully utilized. Tensor core utilization is a better proxy because it reflects how much of the tensor cores are actually being exercised, which rises as image resolution scales up during pre-training. Anyone debugging GPU cluster performance can find more real-world infra lessons like this on daily.dev."}},{"@type":"Question","name":"Why doesn't NVIDIA's DCGM export InfiniBand metrics by default for GPU clusters?","acceptedAnswer":{"@type":"Answer","text":"DCGM exports some NVLink metrics but no InfiniBand metrics out of the box, so teams running large multi-node training clusters have to build custom collection for InfiniBand throughput, wait times, packet counts, and error rates themselves. These metrics turned out critical because most training failures traced back to cross-node communication problems, not the GPUs themselves. Engineers building GPU monitoring pipelines can track similar infra deep dives through daily.dev."}},{"@type":"Question","name":"How can Kubernetes dynamically shift GPU capacity between production inference and model training?","acceptedAnswer":{"@type":"Answer","text":"Kubernetes taints and tolerations combined with Virtual Kubelet can automatically detect when a cluster's GPUs are needed for training and taint the nodes so inference pods get evicted, then use a descheduler to gradually migrate inference pods back once training finishes and GPUs free up. A gradual migration avoids a hard 'no-execute' taint that would kick everything out at once and cause production downtime. Teams balancing GPU costs between training and serving can find similar Kubernetes patterns on daily.dev."}}]} ```