--- title: What Kubernetes Doesn’t Tell You About Running LLMs in... description: Running LLM inference on Kubernetes breaks several assumptions that hold for normal workloads, based on hard lessons from an on-prem deployment. Inference pods... canonical: https://daily.dev/posts/what-kubernetes-doesn-t-tell-you-about-running-llms-in-production-lphan3jww twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: What Kubernetes Doesn’t Tell You About Running LLMs in Production | daily.dev og:description: Running LLM inference on Kubernetes breaks several assumptions that hold for normal workloads, based on hard lessons from an on-prem deployment. Inference pods... og:url: https://daily.dev/posts/what-kubernetes-doesn-t-tell-you-about-running-llms-in-production-lphan3jww og:image: https://api.daily.dev/og/posts/LPhaN3JWW.png og:image:alt: What Kubernetes Doesn’t Tell You About Running LLMs in Production og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # What Kubernetes Doesn’t Tell You About Running LLMs in Production **[Chhoeun K.Keaphirunphirakbot](https://daily.dev/sources/iuklhcqx7w7k6axa5dkxm)** · [@ckphirakbot](https://daily.dev/ckphirakbot) · 11 min read · 12 upvotes · 12 comments ## Summary Running LLM inference on Kubernetes breaks several assumptions that hold for normal workloads, based on hard lessons from an on-prem deployment. Inference pods kept getting evicted under node memory pressure until the team added dedicated tainted GPU node pools and set Guaranteed QoS by matching resource requests and limits. Horizontal Pod Autoscaler doesn't help because loading a model can take close to a minute, so the author recommends vLLM's --max-num-seqs for queueing, vertical tuning, and scheduled manual scaling instead. VRAM usage is unstable due to KV cache growth, so limits need 15-20% headroom plus vLLM's --gpu-memory-utilization capped around 0.85. Zero-downtime rolling updates require maxSurge: 1, maxUnavailable: 0, and a readiness probe hitting vLLM's /health endpoint. ## Content [Support Me: What Kubernetes Doesn’t Tell You About Running LLMs in Production](https://medium.com/@k.keaphirunphirchhoeun/what-kubernetes-doesnt-tell-you-about-running-llms-in-production-30757e6f3b70) What we learned the hard way taking an on-prem LLM from pilot to production. Press enter or click to view image in full size The first time our inference pod got evicted, I assumed it was a configuration mistake. Kubernetes had scheduled it onto a node that already had too much pressure, the node ran out of memory, and the kubelet evicted our model pod to protect the system. Standard behavior. We fixed the resource requests, redeployed, and moved on. Then it happened again. And again. It took us a week to understand what was actually going on — and when we did, it forced us to rethink almost every assumption we’d brought into the project from our experience with conventional workloads. We’d been running Kubernetes for years. We knew how to manage clusters, write resource policies, design deployment strategies. None of it mapped cleanly onto what happens when your pod is holding 27 billion parameters in GPU memory and needs several minutes to restart from scratch. This article is about the gap between general Kubernetes competence and LLM-specific Kubernetes practice. Not the basics — there are plenty of resources for those. This is about the patterns that only become obvious after you’ve been surprised by them in production: the autoscaling strategy that sounds right but isn’t, the resource limit that creates a false sense of safety, the rolling update that turns a five-minute maintenance window into a thirty-minute outage. If your team is moving an LLM workload onto Kubernetes — or has already done it and is running into friction — these are the things I wish someone had told us before we learned them ourselves. Give the Model Its Own Nodes — and Protect Them The eviction problem we kept hitting had a root cause we hadn’t anticipated. We were running inference pods on shared nodes alongside our regular application workloads. The applications were well-behaved under normal load. Under peak load, they weren’t — and when node memory pressure climbed, Kubernetes made the sensible decision to evict lower-priority pods. Our inference pod was lower priority. A model that takes several minutes to reload is not something you want evicted because a batch job got greedy. The fix is dedicated GPU node pools with taints that prevent anything else from being scheduled there. # Taint applied to GPU nodes `kubectl taint nodes nvidia.com/gpu=present:NoSchedule` # Toleration in the inference deployment ``` tolerations: key: "nvidia.com/gpu" operator: "Equal" value: "present" effect: "NoSchedule" nodeSelector: cloud.google.com/gke-accelerator: nvidia-tesla-a100 # or your GPU label ``` With this in place, only pods that explicitly tolerate the taint can land on GPU nodes. No application pods, no batch jobs, no monitoring sidecars that someone forgot to resource-limit. The inference node becomes a controlled environment. But taints only control what gets scheduled there. They don’t control what happens to the inference pod itself under pressure. For that, you need to set the pod’s QoS class to Guaranteed — which requires setting resource requests equal to limits for every container, including the GPU resource: ``` resources: requests: nvidia.com/gpu: "1" memory: "32Gi" limits: nvidia.com/gpu: "1" memory: "32Gi" ``` A Guaranteed QoS pod is the last thing Kubernetes evicts under node pressure, after Burstable and BestEffort pods. Combined with a dedicated tainted node pool, this gives you strong protection against unplanned eviction. We haven’t had an unexpected eviction since making both changes. The caveat: setting memory limits for an inference pod requires you to actually know how much memory the model needs at peak, including KV cache growth under load. Get that number wrong and you’ll trade eviction problems for OOMKill problems. We’ll come back to that. Horizontal Pod Autoscaler Will Not Save You When we started the project, the plan was straightforward: deploy the inference server, configure HPA to scale up under load, and let Kubernetes handle capacity. This is how we manage every other service. It seemed reasonable. It doesn’t work for LLM inference, and understanding why tells you something important about how inference is fundamentally different from application serving. HPA scales by adding pod replicas. For a stateless application, a new replica is ready in seconds and immediately shares the load. For an LLM inference pod, a new replica means loading the model — which on our setup takes the better part of a minute even from local NVMe, and considerably longer from network storage. By the time a scale-out event has loaded a new replica and it’s accepting requests, the latency spike that triggered the scale has long since resolved or caused timeouts. HPA is solving a problem that’s already over. There’s also a more fundamental issue: GPUs are not fungible the way CPUs are. You can’t add half a GPU. If your node pool has two A100s and both are occupied, there is no horizontal scale target available — unless you’ve pre-provisioned spare GPU capacity, which defeats much of the cost argument for autoscaling in the first place. On-premises GPU infrastructure is almost never elastic in the way cloud CPU is. What actually works instead: Concurrency limits and queue depth management. Rather than scaling out reactively, configure your inference server to queue requests up to a defined depth and return a 503 when the queue is full. This is an honest signal that capacity is saturated — one that application teams can handle gracefully with retry logic — rather than silently accumulating requests that will eventually time out. vLLM’s --max-num-seqs parameter controls this directly. Vertical tuning over horizontal scaling. Most of the throughput gains available to an on-premises LLM deployment come from tuning the inference server’s batching and concurrency configuration, not from adding replicas. Continuous batching, optimal batch size for your hardware, and KV cache tuning collectively give you significantly more headroom before you hit a wall than adding a replica that takes a minute to warm up. Manual scale for predictable peaks. If you have predictable traffic patterns — business hours vs. off-hours, end-of-month processing spikes — pre-scale manually on a schedule using a CronJob that updates the Deployment replica count. It’s less elegant than HPA, but it’s honest about the constraints of the workload. The Resource Limit Problem Nobody Talks About Here’s the OOMKill issue I promised we’d come back to — and a broader point about why standard resource management intuitions break down for inference workloads. For a conventional application, setting memory limits is straightforward: profile the application under representative load, add a safety margin, set the limit. The application’s memory usage is relatively stable and predictable. A LLM inference server’s memory usage is not stable. It varies significantly with the length and number of concurrent requests, because the KV cache — the data structure the model uses to avoid recomputing attention across tokens — grows with context length and shrinks when requests complete. Under light load with short prompts, VRAM usage might be 60% of your limit. Under heavy load with long contexts, it can hit 95% and beyond. The practical consequence: if you set your container memory limit based on idle or light-load VRAM usage, you will see OOMKills under real load. And unlike an application OOM that produces a crash with a useful stack trace, an inference OOM often manifests as a CUDA out-of-memory error that terminates a generation mid-stream and leaves the pod in an inconsistent state. The right approach has two parts: First, measure VRAM usage under realistic load, not just at startup. Run your load test — using real input distributions, not short synthetic prompts — and record the peak VRAM usage. That’s your baseline for setting limits. Add at least 15–20% headroom above the observed peak before you set the container memory limit. Second, configure the inference server itself to cap KV cache usage before the OS has to intervene. vLLM’s --gpu-memory-utilization flag (defaulting to 0.90) reserves a fraction of VRAM for the KV cache and refuses to allocate beyond it. Setting this to something like 0.85 gives you a software-level buffer before you approach the hardware limit — meaning vLLM will start rejecting new requests or evicting cache entries gracefully rather than the entire process dying. The combination of a correctly sized container limit and a conservative --gpu-memory-utilization setting gives you two safety nets instead of one. In our experience, it's the difference between an inference server that degrades gracefully under pressure and one that crashes unexpectedly. Zero-Downtime Updates Are Harder Than They Look Rolling updates are one of the things Kubernetes does best. For most workloads, a rolling deployment replaces old pods with new ones gradually, with no gap in service. You’ve done it hundreds of times without thinking about it. For an inference workload, a rolling update has a problem that becomes obvious the first time you do one: the new pod has to load the model before it can serve a single request. During that loading window — which can be a minute or more depending on your storage setup — the new pod is alive, its readiness probe isn’t passing yet, and Kubernetes correctly holds traffic on the old pod. So far so good. But if your Deployment has a single replica (as ours did initially), Kubernetes terminates the old pod before the new one is ready, because that’s what a rolling update with maxUnavailable: 1 and a single replica does. The result is a gap in service proportional to your model load time. This is the pattern that actually works: ``` strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 ``` With maxUnavailable: 0 and maxSurge: 1, Kubernetes brings up the new pod first, waits for it to pass the readiness probe, then terminates the old one. You're briefly running two pods — which means briefly needing two GPUs — but there is no service gap. If you only have one GPU, you can't use this strategy without pre-cordon-ing the node and accepting a maintenance window. The readiness probe is load-bearing. The rollout only works if Kubernetes can accurately determine when the new pod is genuinely ready to serve requests — which means loading is complete, the model is warm, and the inference server is accepting connections. A readiness probe that passes too early (checking only that the HTTP port is open, not that the model is loaded) will send traffic to a pod that isn’t ready. vLLM exposes a /health endpoint that returns 200 only after the model is loaded; use that. ``` readinessProbe: httpGet: path: /health port: 8000 initialDelaySeconds: 30 periodSeconds: 10 failureThreshold: 12 # 2 minutes of retries before marking failed ``` Set initialDelaySeconds to something reasonable — probing every 10 seconds starting at second 5 for a pod that takes 90 seconds to load produces a lot of noise and can cause the rollout to mark the pod failed prematurely. Give it time. What We’re Still Figuring Out I want to be honest about the parts we haven’t solved cleanly, because the internet has enough articles that present everything as figured out. Multi-replica serving with a single GPU node. We currently run a single inference replica because we have one GPU server. This is fine for our current traffic, but it’s a single point of failure: a pod restart means a service interruption of however long model loading takes. We’ve explored time-slicing a single A100 to run two smaller replicas, but GPU time-slicing under bursty inference load introduces latency spikes that we haven’t fully characterized yet. We know the right answer eventually involves a second GPU node; we’re making the case for the hardware. KEDA for inference-aware autoscaling. KEDA (Kubernetes Event-driven Autoscaling) can trigger scale events on custom metrics — including vLLM queue depth — which makes it theoretically better suited for inference than HPA. We’ve started experimenting with it but haven’t promoted it to production yet. The challenge is that scale-out latency (model load time) is still the fundamental constraint, and KEDA doesn’t solve that. It just makes the trigger smarter. Cluster-level GPU resource quotas. We want to enforce per-team limits on GPU usage across a shared cluster so that a new team deploying a large model can’t starve existing services. Kubernetes resource quotas support nvidia.com/gpu as a resource type, but quotas work at the namespace level and our team structure doesn't map cleanly to namespaces. We haven't found an elegant solution yet. If you’re starting this journey now, the single most useful thing you can do is run your load tests before you finalize your Kubernetes configuration, not after. Every mistake I’ve described — the evictions, the OOMKills, the failed rolling updates — was something we could have designed around if we’d had realistic load data earlier. The Kubernetes configuration is downstream of understanding your workload. Understand the workload first. ## Community discussion Top comments from developers on daily.dev. **@mehakulfat** · 2 upvotes > Dedicated GPU node pools and proper scheduling policies are essential for reliable AI inference. Separating workloads prevents unnecessary evictions, reduces downtime, and keeps models running smoothly under peak load. **@diyweb** · 1 upvotes > The rolling update point is the one people underestimate. When model load time dominates, a surge strategy that keeps the old pods serving until new ones pass a real readiness check (model loaded and a warm-up request answered) saves a lot of pain. Did you end up pre-pulling weights onto the GPU nodes, or loading them from shared storage at startup? **@trevorsuna** · 1 upvotes > Dedicated tainted GPU pools, Guaranteed QoS, and VRAM headroom address three different failure modes, which is easy to miss. I’d also measure model-load time and queue depth during rollouts; maxSurge only helps when the cluster has enough spare GPU capacity to start the replacement pod. **@engineerx0** · 1 upvotes > AI generated. Image is for sure from GPT. Username has "bot" in it. **@onlyread** · 1 upvotes > How well did your load tests predict production memory usage? Did real users bring longer prompts or different concurrency patterns than you expected? ## Similar posts on daily.dev - [Running LLM Inference on Kubernetes: What It Actually Takes](https://daily.dev/posts/running-llm-inference-on-kubernetes-what-it-actually-takes-opl6iozrg) · Fairwinds Blog · 0 upvotes · 0 comments - [LLM Inference Cost Optimization: Run AI Inference for Less](https://daily.dev/posts/llm-inference-cost-optimization-run-ai-inference-for-less-syqrz0vbv) · Cast AI · 0 upvotes · 0 comments - [The Inference Bottleneck: Architecting Kubernetes Autoscaling for Production LLMs](https://daily.dev/posts/the-inference-bottleneck-architecting-kubernetes-autoscaling-for-production-llms-3igbdeoqq) · Container Journal · 1 upvotes · 0 comments - [How to deploy and benchmark vLLM with GuideLLM on Kubernetes](https://daily.dev/posts/how-to-deploy-and-benchmark-vllm-with-guidellm-on-kubernetes-sdxhl6tsh) · Red Hat Developer · 1 upvotes · 0 comments - [vLLM Production Deployment: Complete 2026 Guide](https://daily.dev/posts/vllm-production-deployment-complete-2026-guide-c4sdwn4tt) · SitePoint · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#ai-inference](https://daily.dev/tags/ai-inference), [#mlops](https://daily.dev/tags/mlops), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/what-kubernetes-doesn-t-tell-you-about-running-llms-in-production-lphan3jww) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/what-kubernetes-doesn-t-tell-you-about-running-llms-in-production-lphan3jww","headline":"What Kubernetes Doesn’t Tell You About Running LLMs in Production","text":"Running LLM inference on Kubernetes breaks several assumptions that hold for normal workloads, based on hard lessons from an on-prem deployment. Inference pods kept getting evicted under node memory pressure until the team added dedicated tainted GPU node pools and set Guaranteed QoS by matching resource requests and limits. Horizontal Pod Autoscaler doesn't help because loading a model can take close to a minute, so the author recommends vLLM's --max-num-seqs for queueing, vertical tuning, and scheduled manual scaling instead. VRAM usage is unstable due to KV cache growth, so limits need 15-20% headroom plus vLLM's --gpu-memory-utilization capped around 0.85. Zero-downtime rolling updates require maxSurge: 1, maxUnavailable: 0, and a readiness probe hitting vLLM's /health endpoint.","url":"https://daily.dev/posts/what-kubernetes-doesn-t-tell-you-about-running-llms-in-production-lphan3jww","datePublished":"2026-10-29T03:37:09.385Z","dateModified":"2026-10-29T03:37:56.055Z","author":{"@type":"Person","name":"Chhoeun K.Keaphirunphirakbot","url":"https://daily.dev/ckphirakbot","image":"https://lh3.googleusercontent.com/a/AATXAJyu3svre68RXAs8yXKtzZsd-K-S7yMa4mOqbq1m=s100","description":"I am just a DevOps guy","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":2400}},"image":"https://media.daily.dev/image/upload/s--4Txq5IPa--/f_auto,pg_1/v1793245029/posts/LPhaN3JWW?_a=BAMAMiiw0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":12},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":12}],"comment":[{"@type":"Comment","text":"Dedicated GPU node pools and proper scheduling policies are essential for reliable AI inference. Separating workloads prevents unnecessary evictions, reduces downtime, and keeps models running smoothly under peak load.","datePublished":"2026-10-30T05:13:00.277Z","url":"https://daily.dev/posts/LPhaN3JWW#c-Ak6yhHN3N","author":{"@type":"Person","name":"Mehak Ulfat","url":"https://daily.dev/mehakulfat","image":"https://lh3.googleusercontent.com/a/ACg8ocL_fsXZhdOFAKx7AdI7pUJBUg-eKU06JcQaFuvSpwKLS-jbT_I=s96-c"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"The rolling update point is the one people underestimate. When model load time dominates, a surge strategy that keeps the old pods serving until new ones pass a real readiness check (model loaded and a warm-up request answered) saves a lot of pain. Did you end up pre-pulling weights onto the GPU nodes, or loading them from shared storage at startup?","datePublished":"2026-10-31T16:22:14.958Z","url":"https://daily.dev/posts/LPhaN3JWW#c-sY0Jx7m7V","author":{"@type":"Person","name":"DiyWeb","url":"https://daily.dev/diyweb","image":"https://media.daily.dev/image/upload/s--dgS6jDeN--/f_auto/v1790879420/avatars/avatar_774zDwMaRTNS1fv5OKStX?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"Dedicated tainted GPU pools, Guaranteed QoS, and VRAM headroom address three different failure modes, which is easy to miss. I’d also measure model-load time and queue depth during rollouts; maxSurge only helps when the cluster has enough spare GPU capacity to start the replacement pod.","datePublished":"2026-11-01T09:47:33.296Z","url":"https://daily.dev/posts/LPhaN3JWW#c-seaefaZyp","author":{"@type":"Person","name":"Trevor Suna","url":"https://daily.dev/trevorsuna","image":"https://media.daily.dev/image/upload/s--dZ7gXxpp--/f_auto/v1784081551/avatars/avatar_EMoP47rpuw8DNjhp6R1b6?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"AI generated. Image is for sure from GPT. Username has “bot” in it.","datePublished":"2026-10-30T08:29:05.773Z","url":"https://daily.dev/posts/LPhaN3JWW#c-LLmpYCBKi","author":{"@type":"Person","name":"Engineer ","url":"https://daily.dev/engineerx0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"How well did your load tests predict production memory usage? Did real users bring longer prompts or different concurrency patterns than you expected?","datePublished":"2026-10-30T13:51:05.694Z","url":"https://daily.dev/posts/LPhaN3JWW#c-rclLC8CQC","author":{"@type":"Person","name":"OnlyRead","url":"https://daily.dev/onlyread","image":"https://media.daily.dev/image/upload/s--tie3NwCL--/f_auto,pg_1/v1793569075/avatars/avatar_aMdscDAy0fCxDJbh829eu?_a=BAMAMiiw0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/iuklhcqx7w7k6axa5dkxm","name":"Chhoeun K.Keaphirunphirakbot"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Chhoeun K.Keaphirunphirakbot","item":"https://daily.dev/sources/iuklhcqx7w7k6axa5dkxm"},{"@type":"ListItem","position":3,"name":"What Kubernetes Doesn’t Tell You About Running LLMs in Production"}]} ```