--- title: Beyond GPU Utilization: Building an End to End... description: GPU utilization alone is a misleading metric for LLM inference workloads. A GPU at 98% utilization may still be performing poorly if it's stalled on memory... canonical: https://daily.dev/posts/beyond-gpu-utilization-building-an-end-to-end-observability-stack-for-llm-inference-on-kubernetes-tjcterg7r twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Beyond GPU Utilization: Building an End to End Observability Stack for LLM Inference on Kubernetes | daily.dev og:description: GPU utilization alone is a misleading metric for LLM inference workloads. A GPU at 98% utilization may still be performing poorly if it's stalled on memory... og:url: https://daily.dev/posts/beyond-gpu-utilization-building-an-end-to-end-observability-stack-for-llm-inference-on-kubernetes-tjcterg7r og:image: https://api.daily.dev/og/posts/tjcterg7R.png og:image:alt: Beyond GPU Utilization: Building an End to End Observability Stack for LLM Inference on Kubernetes og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Beyond GPU Utilization: Building an End to End Observability Stack for LLM Inference on Kubernetes **[Medium](https://daily.dev/sources/medium_js)** · 12 min read · 0 upvotes · 0 comments ## Summary GPU utilization alone is a misleading metric for LLM inference workloads. A GPU at 98% utilization may still be performing poorly if it's stalled on memory transfers rather than executing tensor computations. The solution is a two-layer observability stack on Kubernetes that correlates hardware telemetry (via NVIDIA DCGM Exporter) with model-level metrics (from vLLM/NVIDIA NIM Prometheus endpoints), both visualized in Grafana. Key GPU metrics covered include GPU utilization, framebuffer memory usage, Tensor Core utilization (DCGM_FI_PROF_PIPE_TENSOR_ACTIVE), and memory copy utilization. Tensor Core utilization is highlighted as the most valuable single metric — two GPUs at 95% utilization can have wildly different Tensor Core activity, revealing whether the bottleneck is compute or memory bandwidth. The article also covers how this correlated view enables data-driven tuning of dynamic batch size and tensor parallelism rather than guesswork. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does my GPU show 98% utilization but tokens per second is still low during LLM inference? High GPU utilization only means the GPU is not idle, not that it is doing productive work. The GPU may be stalled waiting on memory transfers rather than executing matrix multiplications. Checking Tensor Core utilization (DCGM_FI_PROF_PIPE_TENSOR_ACTIVE) alongside memory copy utilization reveals whether the bottleneck is compute or memory bandwidth. _daily.dev surfaces practical GPU tuning insights for engineers debugging LLM inference bottlenecks._ ### Which NVIDIA DCGM metrics matter most for monitoring LLM inference on Kubernetes? The most useful metrics are GPU Utilization (DCGM_FI_DEV_GPU_UTIL), Frame Buffer Memory (DCGM_FI_DEV_FB_USED), Tensor Core Utilization (DCGM_FI_PROF_PIPE_TENSOR_ACTIVE), and Memory Copy Utilization (DCGM_FI_DEV_MEM_COPY_UTIL). Tensor Core utilization is considered especially important because transformer models spend most of their time on matrix operations, making it a better efficiency indicator than raw GPU utilization. _Engineers building GPU observability stacks track metric breakdowns like these on daily.dev._ ### How do I deploy NVIDIA DCGM Exporter on Kubernetes with Prometheus Operator integration? Install it via NVIDIA's official Helm chart using 'helm install dcgm-exporter nvdp/dcgm-exporter --namespace monitoring --create-namespace --set serviceMonitor.enabled=true'. The serviceMonitor.enabled=true flag lets the Prometheus Operator automatically discover the new scrape targets without manually editing Prometheus configuration, after which Prometheus begins collecting GPU telemetry from every NVIDIA GPU in the cluster. _Teams standing up GPU monitoring pipelines find setup walkthroughs like this on daily.dev._ ## Similar posts on daily.dev - [From GPU Silicon to Business Metrics: The 8 Layers of GPU Observability](https://daily.dev/posts/from-gpu-silicon-to-business-metrics-the-8-layers-of-gpu-observability-drqkmcg3z) · Last9 · 0 upvotes · 0 comments - [Get Real-Time Visibility into GPU Usage Across Kubernetes Clusters](https://daily.dev/posts/get-real-time-visibility-into-gpu-usage-across-kubernetes-clusters-ccg90khzv) · NVIDIA Developer · 3 upvotes · 0 comments - [The Inference Bottleneck: Architecting Kubernetes Autoscaling for Production LLMs](https://daily.dev/posts/the-inference-bottleneck-architecting-kubernetes-autoscaling-for-production-llms-3igbdeoqq) · Container Journal · 1 upvotes · 0 comments - [Autoscaling endpoints for LLM inference](https://daily.dev/posts/autoscaling-endpoints-for-llm-inference-ddy21ncoc) · Together AI · 0 upvotes · 0 comments - [How to Choose Full-Stack Observability for NVIDIA AI Factories](https://daily.dev/posts/how-to-choose-full-stack-observability-for-nvidia-ai-factories-lczkkjdar) · NVIDIA Developer · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#observability](https://daily.dev/tags/observability), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/beyond-gpu-utilization-building-an-end-to-end-observability-stack-for-llm-inference-on-kubernetes-tjcterg7r) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Beyond GPU Utilization: Building an End to End Observability Stack for LLM Inference on Kubernetes","url":"https://daily.dev/posts/beyond-gpu-utilization-building-an-end-to-end-observability-stack-for-llm-inference-on-kubernetes-tjcterg7r","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/beyond-gpu-utilization-building-an-end-to-end-observability-stack-for-llm-inference-on-kubernetes-tjcterg7r"},"datePublished":"2026-07-15T07:31:28.858Z","dateModified":"2026-09-13T19:27:09.022Z","description":"GPU utilization alone is a misleading metric for LLM inference workloads. A GPU at 98% utilization may still be performing poorly if it's stalled on memory...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/358c9d1b98a7aae2f8b8365fe1bf79ad?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/358c9d1b98a7aae2f8b8365fe1bf79ad?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/beyond-gpu-utilization-building-an-end-to-end-observability-stack-for-llm-inference-on-kubernetes-tjcterg7r","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,observability,ai-inference,vllm","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"Beyond GPU Utilization: Building an End to End Observability Stack for LLM Inference on Kubernetes"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/beyond-gpu-utilization-building-an-end-to-end-observability-stack-for-llm-inference-on-kubernetes-tjcterg7r#faq","mainEntity":[{"@type":"Question","name":"Why does my GPU show 98% utilization but tokens per second is still low during LLM inference?","acceptedAnswer":{"@type":"Answer","text":"High GPU utilization only means the GPU is not idle, not that it is doing productive work. The GPU may be stalled waiting on memory transfers rather than executing matrix multiplications. Checking Tensor Core utilization (DCGM_FI_PROF_PIPE_TENSOR_ACTIVE) alongside memory copy utilization reveals whether the bottleneck is compute or memory bandwidth. daily.dev surfaces practical GPU tuning insights for engineers debugging LLM inference bottlenecks."}},{"@type":"Question","name":"Which NVIDIA DCGM metrics matter most for monitoring LLM inference on Kubernetes?","acceptedAnswer":{"@type":"Answer","text":"The most useful metrics are GPU Utilization (DCGM_FI_DEV_GPU_UTIL), Frame Buffer Memory (DCGM_FI_DEV_FB_USED), Tensor Core Utilization (DCGM_FI_PROF_PIPE_TENSOR_ACTIVE), and Memory Copy Utilization (DCGM_FI_DEV_MEM_COPY_UTIL). Tensor Core utilization is considered especially important because transformer models spend most of their time on matrix operations, making it a better efficiency indicator than raw GPU utilization. Engineers building GPU observability stacks track metric breakdowns like these on daily.dev."}},{"@type":"Question","name":"How do I deploy NVIDIA DCGM Exporter on Kubernetes with Prometheus Operator integration?","acceptedAnswer":{"@type":"Answer","text":"Install it via NVIDIA's official Helm chart using 'helm install dcgm-exporter nvdp/dcgm-exporter --namespace monitoring --create-namespace --set serviceMonitor.enabled=true'. The serviceMonitor.enabled=true flag lets the Prometheus Operator automatically discover the new scrape targets without manually editing Prometheus configuration, after which Prometheus begins collecting GPU telemetry from every NVIDIA GPU in the cluster. Teams standing up GPU monitoring pipelines find setup walkthroughs like this on daily.dev."}}]} ```