--- title: Continuous Batching Improves Your P50 and Can Wreck Your... description: A measured comparison of continuous batching vs static (gated) batching for LLM inference, run on a DigitalOcean H200 GPU Droplet with vLLM v0.24.0 and Llama... canonical: https://daily.dev/posts/continuous-batching-improves-your-p50-and-can-wreck-your-p99-the-measured-tradeoff-shy2bluhp twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Continuous Batching Improves Your P50 and Can Wreck Your P99: The Measured Tradeoff | daily.dev og:description: A measured comparison of continuous batching vs static (gated) batching for LLM inference, run on a DigitalOcean H200 GPU Droplet with vLLM v0.24.0 and Llama... og:url: https://daily.dev/posts/continuous-batching-improves-your-p50-and-can-wreck-your-p99-the-measured-tradeoff-shy2bluhp og:image: https://api.daily.dev/og/posts/SHY2BlUHp.png og:image:alt: Continuous Batching Improves Your P50 and Can Wreck Your P99: The Measured Tradeoff og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Continuous Batching Improves Your P50 and Can Wreck Your P99: The Measured Tradeoff **[DigitalOcean Community](https://daily.dev/sources/do_community)** · 31 min read · 1 upvotes · 1 comments ## Summary A measured comparison of continuous batching vs static (gated) batching for LLM inference, run on a DigitalOcean H200 GPU Droplet with vLLM v0.24.0 and Llama 3.1 8B. The experiment shows continuous batching dramatically improves median TTFT (8x better than gated at p50) but shifts latency variance into the token stream as inter-token gaps. At 10 req/s, median worst inter-token gap hits 129.6 ms. Disabling chunked prefill widens the p99 worst-gap from 189.9 ms to 267.8 ms (1.41x), confirming chunked prefill's role in moderating stream stalls. Preemptions were zero throughout due to the H200's large KV cache headroom with an 8B model. The piece includes full harness code, raw JSON results, and a reproducibility checklist. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Does disabling chunked prefill in vLLM 0.24.0 make tail latency dramatically worse for LLM inference? Only modestly at moderate load. At 5 requests per second, disabling chunked prefill barely changed p99 worst inter-token gap (134.3ms vs 134.9ms with it on). At 10 requests per second the gap widened from 189.9ms to 267.8ms, a 1.41x increase, without changing time-to-first-token. On an H200 running an 8B model, this did not recreate the severe stall scenario often described in folklore. _Anyone tuning vLLM chunked prefill settings can track measured tradeoffs like these on daily.dev._ ### How much worse is time-to-first-token with gated batch-of-16 admission compared to continuous batching in vLLM? Gated admission with batch size 16 produced a median time-to-first-token of 195.8ms and a p99 of 637.2ms, versus 24.1ms median and 252.2ms p99 for continuous batching at 10 requests per second on the same vLLM 0.24.0 server. That makes gated roughly 8x worse at the median and 2.5x worse at p99, because early requests wait for the whole batch to fill and finish before the next batch starts. _Teams choosing between admission strategies for LLM serving can compare measured numbers like these on daily.dev._ ### Does continuous batching in vLLM cause KV cache preemptions on an H200 GPU running an 8B model? No preemptions occurred across any test rate from 1 to 20 requests per second on an NVIDIA H200 with 141GB memory running Llama 3.1 8B in BF16. The vllm:num_preemptions_total counter stayed at zero throughout, indicating the KV cache headroom was large enough for this model size and trace mix; preemption becomes more likely with larger models, longer context, or higher max_num_seqs. _Engineers debugging LLM serving tail latency can watch for preemption-related findings like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@kartiknvj** · 0 upvotes > The insight that continuous batching does not remove latency variance but relocates it into inter-token gaps is the kind of thing a p50 dashboard will never show you. Median TTFT looking 8x better while p99 stream stalls widen to 267ms means a user feels a fast start and then a stutter mid-response. Did you find the worst inter-token gaps correlated with specific concurrency levels, or were they scattered enough that only per-request traces caught them? ## Similar posts on daily.dev - [Continuous Batching vs. Static Batching in LLM Inference](https://daily.dev/posts/continuous-batching-vs-static-batching-in-llm-inference-xygdguiaz) · DigitalOcean Community · 0 upvotes · 0 comments - [Continuous Batching in LLMs](https://daily.dev/posts/continuous-batching-in-llms-yrzppfm2w) · Daily Dose of Data Science \| Avi Chawla \| Substack · 1 upvotes · 0 comments - [Static vs. Dynamic vs. Continuous Batching in LLMs, clearly explained\!](https://daily.dev/posts/static-vs-dynamic-vs-continuous-batching-in-llms-clearly-explained--qbepnmqas) · Daily Dose of Data Science \| Avi Chawla \| Substack · 0 upvotes · 0 comments - [Continuous batching achieves high GPU utilization for large language models](https://daily.dev/posts/continuous-batching-achieves-high-gpu-utilization-for-large-language-models-z6b15ng2x) · Arpit Bhayani · 0 upvotes · 0 comments - [Why your vLLM p99 latency blows up in production and how chunked prefill and scheduling fix it](https://daily.dev/posts/why-your-vllm-p99-latency-blows-up-in-production-and-how-chunked-prefill-and-scheduling-fix-it-g8kfwibp2) · DigitalOcean Community · 0 upvotes · 0 comments --- Tags: [#gpu](https://daily.dev/tags/gpu), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/continuous-batching-improves-your-p50-and-can-wreck-your-p99-the-measured-tradeoff-shy2bluhp) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Continuous Batching Improves Your P50 and Can Wreck Your P99: The Measured Tradeoff","url":"https://daily.dev/posts/continuous-batching-improves-your-p50-and-can-wreck-your-p99-the-measured-tradeoff-shy2bluhp","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/continuous-batching-improves-your-p50-and-can-wreck-your-p99-the-measured-tradeoff-shy2bluhp"},"datePublished":"2026-08-07T17:42:45.349Z","dateModified":"2026-09-13T20:57:09.574Z","description":"A measured comparison of continuous batching vs static (gated) batching for LLM inference, run on a DigitalOcean H200 GPU Droplet with vLLM v0.24.0 and Llama...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7742157ec3e7212b4ea79ccf0b76be11?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7742157ec3e7212b4ea79ccf0b76be11?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"DigitalOcean Community","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"DigitalOcean Community","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/c1b9d07730e34ea388c39a498a753d6c","url":"https://daily.dev/sources/do_community"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/continuous-batching-improves-your-p50-and-can-wreck-your-p99-the-measured-tradeoff-shy2bluhp","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"gpu,ai-inference,vllm","timeRequired":"PT31M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DigitalOcean Community","item":"https://daily.dev/sources/do_community"},{"@type":"ListItem","position":3,"name":"Continuous Batching Improves Your P50 and Can Wreck Your P99: The Measured Tradeoff"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/continuous-batching-improves-your-p50-and-can-wreck-your-p99-the-measured-tradeoff-shy2bluhp","comment":[{"@type":"Comment","text":"The insight that continuous batching does not remove latency variance but relocates it into inter-token gaps is the kind of thing a p50 dashboard will never show you. Median TTFT looking 8x better while p99 stream stalls widen to 267ms means a user feels a fast start and then a stutter mid-response. Did you find the worst inter-token gaps correlated with specific concurrency levels, or were they scattered enough that only per-request traces caught them?","datePublished":"2026-08-07T19:26:27.200Z","url":"https://daily.dev/posts/SHY2BlUHp#c-PIpYOStU0","author":{"@type":"Person","name":"kartik-nvjk","url":"https://daily.dev/kartiknvj","image":"https://media.daily.dev/image/upload/s--3gGgsVCw--/f_auto/v1781456774/avatars/avatar_TvTVeiMdkRCqWUDullFmy?_a=BAMAMiWQ0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/continuous-batching-improves-your-p50-and-can-wreck-your-p99-the-measured-tradeoff-shy2bluhp#faq","mainEntity":[{"@type":"Question","name":"Does disabling chunked prefill in vLLM 0.24.0 make tail latency dramatically worse for LLM inference?","acceptedAnswer":{"@type":"Answer","text":"Only modestly at moderate load. At 5 requests per second, disabling chunked prefill barely changed p99 worst inter-token gap (134.3ms vs 134.9ms with it on). At 10 requests per second the gap widened from 189.9ms to 267.8ms, a 1.41x increase, without changing time-to-first-token. On an H200 running an 8B model, this did not recreate the severe stall scenario often described in folklore. Anyone tuning vLLM chunked prefill settings can track measured tradeoffs like these on daily.dev."}},{"@type":"Question","name":"How much worse is time-to-first-token with gated batch-of-16 admission compared to continuous batching in vLLM?","acceptedAnswer":{"@type":"Answer","text":"Gated admission with batch size 16 produced a median time-to-first-token of 195.8ms and a p99 of 637.2ms, versus 24.1ms median and 252.2ms p99 for continuous batching at 10 requests per second on the same vLLM 0.24.0 server. That makes gated roughly 8x worse at the median and 2.5x worse at p99, because early requests wait for the whole batch to fill and finish before the next batch starts. Teams choosing between admission strategies for LLM serving can compare measured numbers like these on daily.dev."}},{"@type":"Question","name":"Does continuous batching in vLLM cause KV cache preemptions on an H200 GPU running an 8B model?","acceptedAnswer":{"@type":"Answer","text":"No preemptions occurred across any test rate from 1 to 20 requests per second on an NVIDIA H200 with 141GB memory running Llama 3.1 8B in BF16. The vllm:num_preemptions_total counter stayed at zero throughout, indicating the KV cache headroom was large enough for this model size and trace mix; preemption becomes more likely with larger models, longer context, or higher max_num_seqs. Engineers debugging LLM serving tail latency can watch for preemption-related findings like this on daily.dev."}}]} ```