--- title: Why your vLLM p99 latency blows up in production and how... description: P99 latency spikes in vLLM production deployments are caused by long prefills blocking ongoing decode operations — a scheduling problem, not a hardware one.... canonical: https://daily.dev/posts/why-your-vllm-p99-latency-blows-up-in-production-and-how-chunked-prefill-and-scheduling-fix-it-g8kfwibp2 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why your vLLM p99 latency blows up in production and how chunked prefill and scheduling fix it | daily.dev og:description: P99 latency spikes in vLLM production deployments are caused by long prefills blocking ongoing decode operations — a scheduling problem, not a hardware one.... og:url: https://daily.dev/posts/why-your-vllm-p99-latency-blows-up-in-production-and-how-chunked-prefill-and-scheduling-fix-it-g8kfwibp2 og:image: https://api.daily.dev/og/posts/G8kFwibP2.png og:image:alt: Why your vLLM p99 latency blows up in production and how chunked prefill and scheduling fix it og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why your vLLM p99 latency blows up in production and how chunked prefill and scheduling fix it **[DigitalOcean Community](https://daily.dev/sources/do_community)** · 14 min read · 0 upvotes · 0 comments ## Summary P99 latency spikes in vLLM production deployments are caused by long prefills blocking ongoing decode operations — a scheduling problem, not a hardware one. Continuous batching helps but doesn't fully solve it because prefill still runs as one big unit per iteration. Chunked prefill fixes this by splitting long prefills into smaller token chunks that interleave with decodes, keeping token streams smooth. The Sarathi-Serve research showed this can enable up to 6.9x more traffic within the same latency target. Key tuning knobs include `max_num_batched_tokens` (chunk size dial) and `max_num_seqs`. Scheduling policies (FCFS, priority-based, SLA-aware) and preemption behavior also heavily influence tail latency. Benchmarks using uniform synthetic load miss these effects — testing against real bursty traces like BurstGPT or Azure LLM Inference Traces is essential. The post also clarifies how PagedAttention (KV cache memory management) and chunked prefill interact, and when to use prefill disaggregation instead. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Continuous Batching Improves Your P50 and Can Wreck Your P99: The Measured Tradeoff](https://daily.dev/posts/continuous-batching-improves-your-p50-and-can-wreck-your-p99-the-measured-tradeoff-shy2bluhp) · DigitalOcean Community · 1 upvotes · 1 comments - [The Hidden Bottlenecks in LLM Inference and How to Fix Them](https://daily.dev/posts/the-hidden-bottlenecks-in-llm-inference-and-how-to-fix-them-q1fibdwak) · DigitalOcean Community · 0 upvotes · 0 comments - [The GPU Is Not Your Time-to-First-Token Bottleneck](https://daily.dev/posts/the-gpu-is-not-your-time-to-first-token-bottleneck-gjivhoj0l) · DigitalOcean Community · 1 upvotes · 0 comments - [Static vs. Dynamic vs. Continuous Batching in LLMs, clearly explained\!](https://daily.dev/posts/static-vs-dynamic-vs-continuous-batching-in-llms-clearly-explained--qbepnmqas) · Daily Dose of Data Science \| Avi Chawla \| Substack · 0 upvotes · 0 comments - [Continuous Batching in LLMs](https://daily.dev/posts/continuous-batching-in-llms-yrzppfm2w) · Daily Dose of Data Science \| Avi Chawla \| Substack · 1 upvotes · 0 comments --- Tags: [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/why-your-vllm-p99-latency-blows-up-in-production-and-how-chunked-prefill-and-scheduling-fix-it-g8kfwibp2) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Why your vLLM p99 latency blows up in production and how chunked prefill and scheduling fix it","url":"https://daily.dev/posts/why-your-vllm-p99-latency-blows-up-in-production-and-how-chunked-prefill-and-scheduling-fix-it-g8kfwibp2","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/why-your-vllm-p99-latency-blows-up-in-production-and-how-chunked-prefill-and-scheduling-fix-it-g8kfwibp2"},"datePublished":"2026-07-17T15:56:17.371Z","dateModified":"2026-07-17T16:04:21.489Z","description":"P99 latency spikes in vLLM production deployments are caused by long prefills blocking ongoing decode operations — a scheduling problem, not a hardware one....","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c6bb65789b5513621a96a28f128f3a2a?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c6bb65789b5513621a96a28f128f3a2a?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"DigitalOcean Community","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"DigitalOcean Community","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/c1b9d07730e34ea388c39a498a753d6c","url":"https://daily.dev/sources/do_community"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/why-your-vllm-p99-latency-blows-up-in-production-and-how-chunked-prefill-and-scheduling-fix-it-g8kfwibp2","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-inference,vllm","timeRequired":"PT14M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DigitalOcean Community","item":"https://daily.dev/sources/do_community"},{"@type":"ListItem","position":3,"name":"Why your vLLM p99 latency blows up in production and how chunked prefill and scheduling fix it"}]} ```