--- title: How I massively improved my AI inference performance... description: A deep dive into optimizing LLM inference on a 16x NVIDIA H200 GPU cluster running vLLM via llm-d on Kubernetes, achieving a 71% reduction in... canonical: https://daily.dev/posts/how-i-massively-improved-my-ai-inference-performance-without-buying-new-hardware-jgyj1odfm twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How I massively improved my AI inference performance without buying new hardware | daily.dev og:description: A deep dive into optimizing LLM inference on a 16x NVIDIA H200 GPU cluster running vLLM via llm-d on Kubernetes, achieving a 71% reduction in... og:url: https://daily.dev/posts/how-i-massively-improved-my-ai-inference-performance-without-buying-new-hardware-jgyj1odfm og:image: https://api.daily.dev/og/posts/JgyJ1ODFm.png og:image:alt: How I massively improved my AI inference performance without buying new hardware og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How I massively improved my AI inference performance without buying new hardware **[Red Hat Developer](https://daily.dev/sources/rhdev)** · 23 min read · 0 upvotes · 0 comments ## Summary A deep dive into optimizing LLM inference on a 16x NVIDIA H200 GPU cluster running vLLM via llm-d on Kubernetes, achieving a 71% reduction in time-to-first-token (TTFT) — from 995ms down to 287ms — without any hardware changes. Key levers included tuning gpu_memory_utilization (0.90 to 0.95), computing a model-specific max_num_seqs formula (192 for Qwen3-32B vs 1,433 for Llama-70B-FP8), setting block_size=128 for long-prompt prefix caching, choosing between aggregated vs prefill-decode (PD) disaggregated architectures, and tuning EPP (endpoint picker) routing weights toward a cache_optimized preset (5:1:2). The piece also covers a regression when upgrading from Red Hat AI Inference 3.3 to 3.4 caused by new prefix-cache-aware routing, and how re-tuning recovered and surpassed prior performance. The author built and open-sourced an automation tool, ServeIt Studio, that runs a 12-step pipeline to replicate this tuning process automatically. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What max_num_seqs value should I use for Qwen3-32B versus Llama-70B-FP8 at TP=8 on vLLM? Qwen3-32B at TP=8 should use max_num_seqs=192, while Llama-70B-FP8 at the same TP=8 should use max_num_seqs=1,433, a 7.5x difference. Llama uses grouped-query attention with only 8 KV heads, making its per-sequence KV cache far more compact than Qwen's, so activation memory becomes the binding constraint for Qwen while Llama can support far more concurrent sequences. _daily.dev surfaces practical vLLM tuning experience like this for engineers configuring model-specific inference settings._ ### Should I use aggregated or prefill-decode disaggregated architecture for vLLM inference with long prompts? For long-prompt workloads (2,000 input tokens), a well-tuned aggregated 2xTP8 configuration can beat an untuned PD disaggregated setup, reaching 304ms TTFT versus 675ms for default 3P+1D TP4 on Qwen3-32B with 16 H200 GPUs. PD disaggregation generally still yields lower TTFT once both are tuned, but requires RDMA/InfiniBand for KV cache transfer; without it, tuned aggregated deployments remain a strong option. _engineers weighing PD versus aggregated vLLM deployments can track real benchmark tradeoffs on daily.dev._ ### What gpu_memory_utilization setting should I use for vLLM on NVIDIA H200 GPUs? For H200 GPUs, raising gpu_memory_utilization from the default 0.90 to 0.95 reclaims about 7GB per GPU compared to default, or 56GB across 8 GPUs, enough for roughly 40 more concurrent users before hitting memory pressure. The default 0.90 is a conservative setting meant to work safely from 24GB L4 GPUs up to 140GB H200s, leaving unused headroom on higher-memory cards; pushing too high risks out-of-memory errors under peak load. _teams squeezing more throughput from vLLM on H200s can follow tuning benchmarks like this via daily.dev._ ## Similar posts on daily.dev - [LLM Inference Cost Optimization: Run AI Inference for Less](https://daily.dev/posts/llm-inference-cost-optimization-run-ai-inference-for-less-syqrz0vbv) · Cast AI · 0 upvotes · 0 comments - [Intelligent inference scheduling with llm-d on Red Hat AI](https://daily.dev/posts/intelligent-inference-scheduling-with-llm-d-on-red-hat-ai-q8lqp3mss) · Red Hat Developer · 0 upvotes · 0 comments - [How I doubled my GPU efficiency without buying a single new card](https://daily.dev/posts/how-i-doubled-my-gpu-efficiency-without-buying-a-single-new-card-e57gspube) · InfoWorld · 0 upvotes · 0 comments - [The Infrastructure Behind Making Local LLM Agents Actually Useful](https://daily.dev/posts/the-infrastructure-behind-making-local-llm-agents-actually-useful-ff4qnxir1) · Towards Data Science · 1 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/how-i-massively-improved-my-ai-inference-performance-without-buying-new-hardware-jgyj1odfm) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How I massively improved my AI inference performance without buying new hardware","url":"https://daily.dev/posts/how-i-massively-improved-my-ai-inference-performance-without-buying-new-hardware-jgyj1odfm","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-i-massively-improved-my-ai-inference-performance-without-buying-new-hardware-jgyj1odfm"},"datePublished":"2026-09-23T14:03:18.217Z","dateModified":"2026-09-23T14:03:51.937Z","description":"A deep dive into optimizing LLM inference on a 16x NVIDIA H200 GPU cluster running vLLM via llm-d on Kubernetes, achieving a 71% reduction in...","image":"https://media.daily.dev/image/upload/s--1KxV4ohY--/f_auto/v1722860400/public/Placeholder%2007","thumbnailUrl":"https://media.daily.dev/image/upload/s--1KxV4ohY--/f_auto/v1722860400/public/Placeholder%2007","isAccessibleForFree":true,"articleSection":"Red Hat Developer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Red Hat Developer","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/2a8895f3216c4900bdc455585a4ad3e4","url":"https://daily.dev/sources/rhdev"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-i-massively-improved-my-ai-inference-performance-without-buying-new-hardware-jgyj1odfm","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,ai-inference,vllm","timeRequired":"PT23M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Red Hat Developer","item":"https://daily.dev/sources/rhdev"},{"@type":"ListItem","position":3,"name":"How I massively improved my AI inference performance without buying new hardware"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-i-massively-improved-my-ai-inference-performance-without-buying-new-hardware-jgyj1odfm#faq","mainEntity":[{"@type":"Question","name":"What max_num_seqs value should I use for Qwen3-32B versus Llama-70B-FP8 at TP=8 on vLLM?","acceptedAnswer":{"@type":"Answer","text":"Qwen3-32B at TP=8 should use max_num_seqs=192, while Llama-70B-FP8 at the same TP=8 should use max_num_seqs=1,433, a 7.5x difference. Llama uses grouped-query attention with only 8 KV heads, making its per-sequence KV cache far more compact than Qwen's, so activation memory becomes the binding constraint for Qwen while Llama can support far more concurrent sequences. daily.dev surfaces practical vLLM tuning experience like this for engineers configuring model-specific inference settings."}},{"@type":"Question","name":"Should I use aggregated or prefill-decode disaggregated architecture for vLLM inference with long prompts?","acceptedAnswer":{"@type":"Answer","text":"For long-prompt workloads (2,000 input tokens), a well-tuned aggregated 2xTP8 configuration can beat an untuned PD disaggregated setup, reaching 304ms TTFT versus 675ms for default 3P+1D TP4 on Qwen3-32B with 16 H200 GPUs. PD disaggregation generally still yields lower TTFT once both are tuned, but requires RDMA/InfiniBand for KV cache transfer; without it, tuned aggregated deployments remain a strong option. engineers weighing PD versus aggregated vLLM deployments can track real benchmark tradeoffs on daily.dev."}},{"@type":"Question","name":"What gpu_memory_utilization setting should I use for vLLM on NVIDIA H200 GPUs?","acceptedAnswer":{"@type":"Answer","text":"For H200 GPUs, raising gpu_memory_utilization from the default 0.90 to 0.95 reclaims about 7GB per GPU compared to default, or 56GB across 8 GPUs, enough for roughly 40 more concurrent users before hitting memory pressure. The default 0.90 is a conservative setting meant to work safely from 24GB L4 GPUs up to 140GB H200s, leaving unused headroom on higher-memory cards; pushing too high risks out-of-memory errors under peak load. teams squeezing more throughput from vLLM on H200s can follow tuning benchmarks like this via daily.dev."}}]} ```