--- title: "Running LLM Inference on Kubernetes: What It Actually Takes" url: https://daily.dev/posts/running-llm-inference-on-kubernetes-what-it-actually-takes-opl6iozrg source_url: https://www.fairwinds.com/blog/running-llm-inference-on-kubernetes-what-it-actually-takes type: article source: "Fairwinds Blog" published: 2026-06-05T15:13:07.673Z updated: 2026-06-05T15:13:34.345Z tags: ["kubernetes", "gpu", "ai-inference", "vllm"] reading_time: 8 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Running LLM Inference on Kubernetes: What It Actually Takes **[Fairwinds Blog](https://daily.dev/sources/fairwinds-blog)** · 8 min read · 0 upvotes · 0 comments ## Summary Running LLM inference on Kubernetes is fundamentally different from deploying standard web apps. Inference pods carry large state, take 15–30 minutes to start after loading model weights, and require GPU-aware scheduling, custom autoscaling metrics, and careful cost governance. Four cluster prerequisites must be addressed: NVIDIA device plugin setup, gang scheduling for disaggregated deployments, topology-aware pod placement for tensor parallelism, and KEDA-based autoscaling using GPU utilization and queue depth instead of CPU/memory. Three inference tooling options are compared: the vLLM Helm chart (proof-of-concept only, no autoscaling or observability), Ollama (simple but lacks Prometheus metrics and elastic scaling), and KubeAI (production-ready, with model caching via EFS, scale-to-zero, built-in Prometheus metrics, and vLLM as the GPU backend). KubeAI's routing layer enables autoscaling transparency and multi-GPU tensor parallelism. Cost control requires both Karpenter for on-demand node provisioning and RBAC/governance to prevent idle GPU waste. Platform teams should own cluster fundamentals and inference orchestration; application teams own model selection and endpoint consumption. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [LLM Inference Cost Optimization: Run AI Inference for Less](https://daily.dev/posts/llm-inference-cost-optimization-run-ai-inference-for-less-syqrz0vbv) · Cast AI · 0 upvotes · 0 comments - [The Inference Bottleneck: Architecting Kubernetes Autoscaling for Production LLMs](https://daily.dev/posts/the-inference-bottleneck-architecting-kubernetes-autoscaling-for-production-llms-3igbdeoqq) · Container Journal · 1 upvotes · 0 comments - [Running a self-hosted LLM in Kubernetes with vLLM](https://daily.dev/posts/running-a-self-hosted-llm-in-kubernetes-with-vllm-znxvebe4x) · CNCF · 3 upvotes · 1 comments - [Open-weight AI on Kubernetes: A Developer’s Guide to vLLM vs. KubeAI](https://daily.dev/posts/open-weight-ai-on-kubernetes-a-developer-s-guide-to-vllm-vs-kubeai-hdxk8ddoh) · SitePoint · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#gpu](https://daily.dev/tags/gpu), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/running-llm-inference-on-kubernetes-what-it-actually-takes-opl6iozrg)