--- title: Why Your GPU Fails at 3 Users (LLM Inference Isn't a... description: A GPU inference walkthrough demonstrates why serving concurrent users, not fitting model weights, is the real bottleneck for self-hosted LLMs. Using an 8B... canonical: https://daily.dev/posts/why-your-gpu-fails-at-3-users-llm-inference-isn-t-a-compute-problem--qx7xaoaqs twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why Your GPU Fails at 3 Users (LLM Inference Isn't a Compute Problem) | daily.dev og:description: A GPU inference walkthrough demonstrates why serving concurrent users, not fitting model weights, is the real bottleneck for self-hosted LLMs. Using an 8B... og:url: https://daily.dev/posts/why-your-gpu-fails-at-3-users-llm-inference-isn-t-a-compute-problem--qx7xaoaqs og:image: https://api.daily.dev/og/posts/Qx7xaOaqs.png og:image:alt: Why Your GPU Fails at 3 Users (LLM Inference Isn't a Compute Problem) og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why Your GPU Fails at 3 Users (LLM Inference Isn't a Compute Problem) **[DevOps Toolkit](https://daily.dev/sources/devops-toolkit)** · 22 min read · 0 upvotes · 0 comments ## Summary A GPU inference walkthrough demonstrates why serving concurrent users, not fitting model weights, is the real bottleneck for self-hosted LLMs. Using an 8B parameter model on an Nvidia L4 GPU and vLLM, the demonstration shows that KV cache (memory for conversation context) determines how many users can be served concurrently, not model size. With 100 simultaneous requests, the unoptimized deployment supported under 3 concurrent conversations at full context length despite the model fitting comfortably in memory. Applying 8-bit weight quantization tripled capacity by freeing memory for KV cache rather than making computation faster, with negligible quality loss for chat tasks but measurable risk for math, code, and reasoning tasks. Adding KV cache quantization on top doubled capacity again, eventually shifting the bottleneck from memory to raw compute (arithmetic-bound rather than memory-bound). ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does my GPU serve only a few concurrent users even though the LLM model fits comfortably in VRAM? GPU memory left over after loading model weights is consumed by the KV cache, which stores intermediate attention values for every ongoing conversation. This cache grows with both conversation length and the number of simultaneous users, so it (not the model weights) sets the real concurrency ceiling. On a 24GB Nvidia L4 GPU running an 8B parameter model at 16-bit precision, only about 6GB remained for KV cache after weights and engine overhead, supporting under three full-length concurrent conversations at 8,000 tokens each, though about a dozen at shorter context lengths. _Anyone sizing GPU capacity for LLM serving can track memory-management techniques like these on daily.dev._ ### How much does quantizing LLM weights from 16-bit to 8-bit improve concurrent user capacity on a GPU? Quantizing model weights from 16-bit to 8-bit precision on an 8 billion parameter model roughly tripled the concurrency ceiling, because the memory freed by shrinking weights by about a third went entirely into KV cache space, which was the smallest and most constrained slice of GPU memory. Quality loss on chat and instruction-following tasks was below a percent, though multi-step math, scientific reasoning, and code generation are more sensitive to the precision drop. _Developers weighing quantization tradeoffs for their own inference workloads can follow practical breakdowns like this on daily.dev._ ### Should I quantize LLM weights below 4 bits to save GPU memory? No, going below 4 bits is not recommended, especially for agents or tool-calling workloads. Accuracy holds up well at 8 bits (near original quality) and 4 bits still recovers most quality with better quantization methods, but at 2 bits accuracy drops off sharply with no current encoding method fixing it, and reasoning capability is the first casualty below 4 bits. _Teams deciding how far to push quantization for agentic workloads can track guidance like this on daily.dev._ ## Similar posts on daily.dev - [The KV Cache Tax: Why Inference Servers Run Out of Memory Before Compute](https://daily.dev/posts/the-kv-cache-tax-why-inference-servers-run-out-of-memory-before-compute-yzoonznpj) · Towards Data Science · 0 upvotes · 0 comments - [The KV Cache Survival Guide: Why Your GPU Runs Out of Memory with Local LLMs](https://daily.dev/posts/the-kv-cache-survival-guide-why-your-gpu-runs-out-of-memory-with-local-llms-gxyoa0vqe) · SitePoint · 1 upvotes · 0 comments - [How to Choose the Right GPU for vLLM Inference](https://daily.dev/posts/how-to-choose-the-right-gpu-for-vllm-inference-1felswdq3) · DigitalOcean Community · 1 upvotes · 0 comments - [The LLM Inference Trilemma: Throughput, Latency, Cost](https://daily.dev/posts/the-llm-inference-trilemma-throughput-latency-cost-ws3duzsii) · DigitalOcean · 0 upvotes · 0 comments - [Why An LLM’s Memory Gets Expensive and How to Fix It](https://daily.dev/posts/why-an-llm-s-memory-gets-expensive-and-how-to-fix-it-tfmmfttjf) · ByteByteGo · 2 upvotes · 0 comments --- Tags: [#data-science](https://daily.dev/tags/data-science), [#gpu](https://daily.dev/tags/gpu), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/why-your-gpu-fails-at-3-users-llm-inference-isn-t-a-compute-problem--qx7xaoaqs) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Why Your GPU Fails at 3 Users (LLM Inference Isn't a Compute Problem)","url":"https://daily.dev/posts/why-your-gpu-fails-at-3-users-llm-inference-isn-t-a-compute-problem--qx7xaoaqs","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/why-your-gpu-fails-at-3-users-llm-inference-isn-t-a-compute-problem--qx7xaoaqs"},"datePublished":"2026-09-14T16:22:26.337Z","dateModified":"2026-09-14T22:20:14.540Z","description":"A GPU inference walkthrough demonstrates why serving concurrent users, not fitting model weights, is the real bottleneck for self-hosted LLMs. Using an 8B...","image":"https://i.ytimg.com/vi/g_5g1hBmAzA/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/g_5g1hBmAzA/sddefault.jpg","isAccessibleForFree":true,"articleSection":"DevOps Toolkit","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"DevOps Toolkit","logo":"https://media.daily.dev/image/upload/s--VPsieX2o--/f_auto/v1704633572/logos/devops-toolkit.jpg","url":"https://daily.dev/sources/devops-toolkit"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/why-your-gpu-fails-at-3-users-llm-inference-isn-t-a-compute-problem--qx7xaoaqs","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-science,gpu,ai-inference,vllm","timeRequired":"PT22M","video":{"@type":"VideoObject","name":"Why Your GPU Fails at 3 Users (LLM Inference Isn't a Compute Problem)","description":"A GPU inference walkthrough demonstrates why serving concurrent users, not fitting model weights, is the real bottleneck for self-hosted LLMs. Using an 8B...","thumbnailUrl":"https://i.ytimg.com/vi/g_5g1hBmAzA/sddefault.jpg","uploadDate":"2026-09-14T16:22:26.337Z","duration":"PT22M","url":"https://api.daily.dev/r/Qx7xaOaqs","embedUrl":"https://www.youtube.com/embed/g_5g1hBmAzA"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DevOps Toolkit","item":"https://daily.dev/sources/devops-toolkit"},{"@type":"ListItem","position":3,"name":"Why Your GPU Fails at 3 Users (LLM Inference Isn't a Compute Problem)"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/why-your-gpu-fails-at-3-users-llm-inference-isn-t-a-compute-problem--qx7xaoaqs#faq","mainEntity":[{"@type":"Question","name":"Why does my GPU serve only a few concurrent users even though the LLM model fits comfortably in VRAM?","acceptedAnswer":{"@type":"Answer","text":"GPU memory left over after loading model weights is consumed by the KV cache, which stores intermediate attention values for every ongoing conversation. This cache grows with both conversation length and the number of simultaneous users, so it (not the model weights) sets the real concurrency ceiling. On a 24GB Nvidia L4 GPU running an 8B parameter model at 16-bit precision, only about 6GB remained for KV cache after weights and engine overhead, supporting under three full-length concurrent conversations at 8,000 tokens each, though about a dozen at shorter context lengths. Anyone sizing GPU capacity for LLM serving can track memory-management techniques like these on daily.dev."}},{"@type":"Question","name":"How much does quantizing LLM weights from 16-bit to 8-bit improve concurrent user capacity on a GPU?","acceptedAnswer":{"@type":"Answer","text":"Quantizing model weights from 16-bit to 8-bit precision on an 8 billion parameter model roughly tripled the concurrency ceiling, because the memory freed by shrinking weights by about a third went entirely into KV cache space, which was the smallest and most constrained slice of GPU memory. Quality loss on chat and instruction-following tasks was below a percent, though multi-step math, scientific reasoning, and code generation are more sensitive to the precision drop. Developers weighing quantization tradeoffs for their own inference workloads can follow practical breakdowns like this on daily.dev."}},{"@type":"Question","name":"Should I quantize LLM weights below 4 bits to save GPU memory?","acceptedAnswer":{"@type":"Answer","text":"No, going below 4 bits is not recommended, especially for agents or tool-calling workloads. Accuracy holds up well at 8 bits (near original quality) and 4 bits still recovers most quality with better quantization methods, but at 2 bits accuracy drops off sharply with no current encoding method fixing it, and reasoning capability is the first casualty below 4 bits. Teams deciding how far to push quantization for agentic workloads can track guidance like this on daily.dev."}}]} ```