Running LLM inference on Kubernetes is fundamentally different from deploying standard web apps. Inference pods carry large state, take 15–30 minutes to start after loading model weights, and require GPU-aware scheduling, custom autoscaling metrics, and careful cost governance. Four cluster prerequisites must be addressed: NVIDIA device plugin setup, gang scheduling for disaggregated deployments, topology-aware pod placement for tensor parallelism, and KEDA-based autoscaling using GPU utilization and queue depth instead of CPU/memory. Three inference tooling options are compared: the vLLM Helm chart (proof-of-concept only, no autoscaling or observability), Ollama (simple but lacks Prometheus metrics and elastic scaling), and KubeAI (production-ready, with model caching via EFS, scale-to-zero, built-in Prometheus metrics, and vLLM as the GPU backend). KubeAI's routing layer enables autoscaling transparency and multi-GPU tensor parallelism. Cost control requires both Karpenter for on-demand node provisioning and RBAC/governance to prevent idle GPU waste. Platform teams should own cluster fundamentals and inference orchestration; application teams own model selection and endpoint consumption.
