--- title: AI Didn’t Break Kubernetes Optimization. GPUs Did. description: Kubernetes allocates GPUs whole, so inference workloads often hold an entire GPU while using only a fraction of its compute. Time-slicing lets jobs share a GPU... canonical: https://daily.dev/posts/ai-didn-t-break-kubernetes-optimization-gpus-did--wq98xzgiz twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: AI Didn’t Break Kubernetes Optimization. GPUs Did. | daily.dev og:description: Kubernetes allocates GPUs whole, so inference workloads often hold an entire GPU while using only a fraction of its compute. Time-slicing lets jobs share a GPU... og:url: https://daily.dev/posts/ai-didn-t-break-kubernetes-optimization-gpus-did--wq98xzgiz og:image: https://api.daily.dev/og/posts/Wq98xzGiZ.png og:image:alt: AI Didn’t Break Kubernetes Optimization. GPUs Did. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # AI Didn’t Break Kubernetes Optimization. GPUs Did. **[Cloud Native Now](https://daily.dev/sources/cloudnativenow)** · 4 min read · 0 upvotes · 0 comments ## Summary Kubernetes allocates GPUs whole, so inference workloads often hold an entire GPU while using only a fraction of its compute. Time-slicing lets jobs share a GPU but offers no memory isolation, so one spiking job can trigger out-of-memory errors in unrelated workloads. Node-level memory and compute metrics can be misleading too, since engines like vLLM grab memory at startup regardless of traffic. The fix: pull queue depth and other metrics from inside the serving engine to guide autoscaling, keep spiky jobs off shared GPUs, and use MIG hardware partitioning on supported NVIDIA hardware when sharing is unavoidable. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### why do I get out-of-memory errors on a GPU pod that isn't doing anything wrong when sharing a GPU with time-slicing Time-slicing provides no memory isolation between jobs sharing a GPU, so all jobs compete for the same memory pool. If one job spikes in memory usage, it can exhaust the pool and trigger out-of-memory errors in other jobs on that GPU, even ones that behaved normally. The spiking job typically gets blamed while the real cause stays hidden in cross-job memory contention. _Teams debugging shared-GPU memory failures can track Kubernetes GPU optimization patterns like this on daily.dev._ ### why does my vLLM server show high GPU memory usage even when it's idle vLLM and similar serving engines grab a large chunk of GPU memory at startup regardless of incoming traffic, so an idle server and a fully loaded one can look identical on memory metrics. Compute utilization is also unreliable for load assessment because continuous batching obscures useful work versus inefficiency at the node level. Queue depth, tracked inside the serving engine, is a more reliable signal of actual demand. _Anyone tuning vLLM autoscaling can keep up with GPU inference monitoring approaches via daily.dev._ ### should I use time-slicing or MIG to share a GPU across multiple Kubernetes workloads MIG hardware partitioning gives memory isolation that time-slicing lacks, turning a noisy-neighbor risk into a predictable capacity planning decision, though it reduces flexibility since partition sizes are fixed. Time-slicing lets jobs take turns on a GPU but shares one memory pool, so a spiking job can cause out-of-memory errors elsewhere. Spiky, memory-hungry jobs should be kept off shared GPUs entirely. _Engineers weighing GPU sharing strategies can compare Kubernetes MIG versus time-slicing trade-offs on daily.dev._ ## Similar posts on daily.dev - [AI jobs are waiting while GPUs sit idle. Kubernetes has a new plan.](https://daily.dev/posts/ai-jobs-are-waiting-while-gpus-sit-idle-kubernetes-has-a-new-plan--hwksxjnw5) · The New Stack · 0 upvotes · 0 comments - [Stop Treating Your Models Like Microservices](https://daily.dev/posts/stop-treating-your-models-like-microservices-pwqhglvbb) · Cloud Native Now · 0 upvotes · 0 comments - [Kubernetes GPU Optimization: How to Cut GPU Waste Without Slowing Workloads](https://daily.dev/posts/kubernetes-gpu-optimization-how-to-cut-gpu-waste-without-slowing-workloads-pi9fl1mnv) · Cast AI · 0 upvotes · 0 comments - [How to Diagnose and Fix AI Inference Latency on Kubernetes](https://daily.dev/posts/how-to-diagnose-and-fix-ai-inference-latency-on-kubernetes-flymwvtow) · freeCodeCamp · 1 upvotes · 0 comments - [The Inference Bottleneck: Architecting Kubernetes Autoscaling for Production LLMs](https://daily.dev/posts/the-inference-bottleneck-architecting-kubernetes-autoscaling-for-production-llms-3igbdeoqq) · Container Journal · 1 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#observability](https://daily.dev/tags/observability), [#gpu](https://daily.dev/tags/gpu), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/ai-didn-t-break-kubernetes-optimization-gpus-did--wq98xzgiz) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"AI Didn’t Break Kubernetes Optimization. GPUs Did.","url":"https://daily.dev/posts/ai-didn-t-break-kubernetes-optimization-gpus-did--wq98xzgiz","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/ai-didn-t-break-kubernetes-optimization-gpus-did--wq98xzgiz"},"datePublished":"2026-10-26T20:48:09.370Z","dateModified":"2026-10-26T21:00:28.297Z","description":"Kubernetes allocates GPUs whole, so inference workloads often hold an entire GPU while using only a fraction of its compute. Time-slicing lets jobs share a GPU...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/aa7cff38bc67dff78a9748fe9ca6e15c?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/aa7cff38bc67dff78a9748fe9ca6e15c?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Cloud Native Now","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Cloud Native Now","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/8acc1ecd93ce4299b615252f32d78de5","url":"https://daily.dev/sources/cloudnativenow"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/ai-didn-t-break-kubernetes-optimization-gpus-did--wq98xzgiz","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,observability,gpu,ai-inference,vllm","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Cloud Native Now","item":"https://daily.dev/sources/cloudnativenow"},{"@type":"ListItem","position":3,"name":"AI Didn’t Break Kubernetes Optimization. GPUs Did."}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/ai-didn-t-break-kubernetes-optimization-gpus-did--wq98xzgiz#faq","mainEntity":[{"@type":"Question","name":"why do I get out-of-memory errors on a GPU pod that isn't doing anything wrong when sharing a GPU with time-slicing","acceptedAnswer":{"@type":"Answer","text":"Time-slicing provides no memory isolation between jobs sharing a GPU, so all jobs compete for the same memory pool. If one job spikes in memory usage, it can exhaust the pool and trigger out-of-memory errors in other jobs on that GPU, even ones that behaved normally. The spiking job typically gets blamed while the real cause stays hidden in cross-job memory contention. Teams debugging shared-GPU memory failures can track Kubernetes GPU optimization patterns like this on daily.dev."}},{"@type":"Question","name":"why does my vLLM server show high GPU memory usage even when it's idle","acceptedAnswer":{"@type":"Answer","text":"vLLM and similar serving engines grab a large chunk of GPU memory at startup regardless of incoming traffic, so an idle server and a fully loaded one can look identical on memory metrics. Compute utilization is also unreliable for load assessment because continuous batching obscures useful work versus inefficiency at the node level. Queue depth, tracked inside the serving engine, is a more reliable signal of actual demand. Anyone tuning vLLM autoscaling can keep up with GPU inference monitoring approaches via daily.dev."}},{"@type":"Question","name":"should I use time-slicing or MIG to share a GPU across multiple Kubernetes workloads","acceptedAnswer":{"@type":"Answer","text":"MIG hardware partitioning gives memory isolation that time-slicing lacks, turning a noisy-neighbor risk into a predictable capacity planning decision, though it reduces flexibility since partition sizes are fixed. Time-slicing lets jobs take turns on a GPU but shares one memory pool, so a spiking job can cause out-of-memory errors elsewhere. Spiky, memory-hungry jobs should be kept off shared GPUs entirely. Engineers weighing GPU sharing strategies can compare Kubernetes MIG versus time-slicing trade-offs on daily.dev."}}]} ```