--- title: Busy GPUs can hide slow AI inference on Kubernetes description: A fully utilized GPU doesn't mean an AI inference service is performing well, HPE product managers argue in this sponsored article. Teams should measure time... canonical: https://daily.dev/posts/busy-gpus-can-hide-slow-ai-inference-on-kubernetes-iqvdx7rfm twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Busy GPUs can hide slow AI inference on Kubernetes | daily.dev og:description: A fully utilized GPU doesn't mean an AI inference service is performing well, HPE product managers argue in this sponsored article. Teams should measure time... og:url: https://daily.dev/posts/busy-gpus-can-hide-slow-ai-inference-on-kubernetes-iqvdx7rfm og:image: https://api.daily.dev/og/posts/iqVdx7rfM.png og:image:alt: Busy GPUs can hide slow AI inference on Kubernetes og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Busy GPUs can hide slow AI inference on Kubernetes **[The New Stack](https://daily.dev/sources/newstack)** · 8 min read · 0 upvotes · 0 comments ## Summary A fully utilized GPU doesn't mean an AI inference service is performing well, HPE product managers argue in this sponsored article. Teams should measure time to first token, time per output token, reliability, and answer quality separately, then compare live behavior against a pre-launch baseline. GPU utilization alone can't identify bottlenecks like undersized GPU partitions, network or vector-database delays, or misrouted requests. Kubernetes' Horizontal Pod Autoscaler can work for inference if fed the right custom metrics, since CPU utilization is a poor scaling trigger. HPE positions Morpheus as its provisioning engine and OpsRamp as its observability engine, extending telemetry to token rates and time to first token. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why is 100% GPU utilization not a good indicator that my AI inference service is healthy? GPU utilization shows the hardware is busy but says nothing about whether requests are being served well. A model can return its first token quickly but take too long to finish, or respond fast with a wrong answer, while GPU activity looks fully saturated. Bottlenecks can instead come from an undersized GPU partition, slow vector-database responses, or misrouted requests to the wrong model. _daily.dev surfaces practical takes like this for engineers debugging misleading infra metrics on Kubernetes._ ### Can I use Kubernetes Horizontal Pod Autoscaler for scaling AI inference workloads? Yes, the Horizontal Pod Autoscaler can scale inference pods using configured custom or external metrics instead of relying on CPU utilization, which is a poor trigger for inference workloads. Teams still need to supply and test a metrics pipeline, set scaling bounds, and ensure enough accelerator capacity is available, since a scaling policy cannot instantly create more GPU capacity. _developers planning GPU autoscaling strategies can track this kind of Kubernetes guidance on daily.dev._ ### What metrics should I measure to evaluate an LLM inference service before launch? Recommended metrics include time to first token, time per output token, inter-token latency, reliability, capacity, resource efficiency, cost, and answer quality. Profiling should happen before customers depend on the service, testing representative prompt and context lengths, output volumes, and traffic levels to establish a baseline for comparing production behavior later. _teams building a pre-launch checklist for AI services can follow this kind of operational guidance on daily.dev._ ## Similar posts on daily.dev - [The Inference Bottleneck: Architecting Kubernetes Autoscaling for Production LLMs](https://daily.dev/posts/the-inference-bottleneck-architecting-kubernetes-autoscaling-for-production-llms-3igbdeoqq) · Container Journal · 1 upvotes · 0 comments - [AI jobs are waiting while GPUs sit idle. Kubernetes has a new plan.](https://daily.dev/posts/ai-jobs-are-waiting-while-gpus-sit-idle-kubernetes-has-a-new-plan--hwksxjnw5) · The New Stack · 0 upvotes · 0 comments - [AI Didn’t Break Kubernetes Optimization. GPUs Did.](https://daily.dev/posts/ai-didn-t-break-kubernetes-optimization-gpus-did--wq98xzgiz) · Cloud Native Now · 0 upvotes · 0 comments - [Best GPU Optimization Tools for Kubernetes and AI Workloads \(2026\)](https://daily.dev/posts/best-gpu-optimization-tools-for-kubernetes-and-ai-workloads-2026--xv0s5w89i) · Cast AI · 0 upvotes · 0 comments - [How to Diagnose and Fix AI Inference Latency on Kubernetes](https://daily.dev/posts/how-to-diagnose-and-fix-ai-inference-latency-on-kubernetes-flymwvtow) · freeCodeCamp · 1 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#observability](https://daily.dev/tags/observability), [#gpu](https://daily.dev/tags/gpu), [#ai-inference](https://daily.dev/tags/ai-inference) [View this post on daily.dev](https://daily.dev/posts/busy-gpus-can-hide-slow-ai-inference-on-kubernetes-iqvdx7rfm) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Busy GPUs can hide slow AI inference on Kubernetes","url":"https://daily.dev/posts/busy-gpus-can-hide-slow-ai-inference-on-kubernetes-iqvdx7rfm","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/busy-gpus-can-hide-slow-ai-inference-on-kubernetes-iqvdx7rfm"},"datePublished":"2026-10-30T20:02:26.955Z","dateModified":"2026-10-30T20:03:11.606Z","description":"A fully utilized GPU doesn't mean an AI inference service is performing well, HPE product managers argue in this sponsored article. Teams should measure time...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/48b6356c47d54cc252f96abe6222c017?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/48b6356c47d54cc252f96abe6222c017?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"The New Stack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"The New Stack","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/newstack","url":"https://daily.dev/sources/newstack"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/busy-gpus-can-hide-slow-ai-inference-on-kubernetes-iqvdx7rfm","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,observability,gpu,ai-inference","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"The New Stack","item":"https://daily.dev/sources/newstack"},{"@type":"ListItem","position":3,"name":"Busy GPUs can hide slow AI inference on Kubernetes"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/busy-gpus-can-hide-slow-ai-inference-on-kubernetes-iqvdx7rfm#faq","mainEntity":[{"@type":"Question","name":"Why is 100% GPU utilization not a good indicator that my AI inference service is healthy?","acceptedAnswer":{"@type":"Answer","text":"GPU utilization shows the hardware is busy but says nothing about whether requests are being served well. A model can return its first token quickly but take too long to finish, or respond fast with a wrong answer, while GPU activity looks fully saturated. Bottlenecks can instead come from an undersized GPU partition, slow vector-database responses, or misrouted requests to the wrong model. daily.dev surfaces practical takes like this for engineers debugging misleading infra metrics on Kubernetes."}},{"@type":"Question","name":"Can I use Kubernetes Horizontal Pod Autoscaler for scaling AI inference workloads?","acceptedAnswer":{"@type":"Answer","text":"Yes, the Horizontal Pod Autoscaler can scale inference pods using configured custom or external metrics instead of relying on CPU utilization, which is a poor trigger for inference workloads. Teams still need to supply and test a metrics pipeline, set scaling bounds, and ensure enough accelerator capacity is available, since a scaling policy cannot instantly create more GPU capacity. developers planning GPU autoscaling strategies can track this kind of Kubernetes guidance on daily.dev."}},{"@type":"Question","name":"What metrics should I measure to evaluate an LLM inference service before launch?","acceptedAnswer":{"@type":"Answer","text":"Recommended metrics include time to first token, time per output token, inter-token latency, reliability, capacity, resource efficiency, cost, and answer quality. Profiling should happen before customers depend on the service, testing representative prompt and context lengths, output volumes, and traffic levels to establish a baseline for comparing production behavior later. teams building a pre-launch checklist for AI services can follow this kind of operational guidance on daily.dev."}}]} ```