--- title: Continuous Batching in LLMs | daily.dev description: Continuous batching is the scheduling technique used by default in serving engines like vLLM, SGLang, TGI, and TensorRT-LLM to handle LLM inference's... canonical: https://daily.dev/posts/continuous-batching-in-llms-yrzppfm2w twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Continuous Batching in LLMs | daily.dev og:description: Continuous batching is the scheduling technique used by default in serving engines like vLLM, SGLang, TGI, and TensorRT-LLM to handle LLM inference's... og:url: https://daily.dev/posts/continuous-batching-in-llms-yrzppfm2w og:image: https://api.daily.dev/og/posts/YrZPpfM2W.png og:image:alt: Continuous Batching in LLMs og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Continuous Batching in LLMs **[Daily Dose of Data Science \| Avi Chawla \| Substack](https://daily.dev/sources/dailydoseofds)** · 11 min read · 1 upvotes · 0 comments ## Summary Continuous batching is the scheduling technique used by default in serving engines like vLLM, SGLang, TGI, and TensorRT-LLM to handle LLM inference's variable-length decoding. Unlike traditional ML batching, where a fixed batch runs at the pace of its slowest member, continuous batching rebuilds the batch at every forward pass, letting finished requests exit and waiting requests enter immediately. The explanation walks through vLLM's V1 scheduler: budget allocation via max_num_batched_tokens and max_num_seqs, selective batching (flattening tokens for shared operations while splitting for attention), KV block reservation, and preemption behavior when memory runs out. Anyscale's benchmark on OPT-13B showed vLLM reaching 23x the throughput of naive Hugging Face serving on the same A100, while static batching fell to around 81 tokens/s under varied output lengths. Preemption recompute cost and Prometheus's total_cumulative_preemption_cnt metric are highlighted as key diagnostics for latency issues, alongside tuning knobs like gpu_memory_utilization, max_num_seqs, and tensor_parallel_size. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is continuous batching in LLM inference and why does it improve throughput? Continuous batching is a scheduling approach where the batch of requests is rebuilt at every forward pass instead of being fixed until all requests finish, so a finished request leaves immediately and a waiting request takes its slot rather than idling GPU capacity. It solves the problem where a fixed batch runs at the pace of its slowest member since output lengths vary and are unknown in advance. vLLM, SGLang, TGI, and TensorRT-LLM all use it by default. _Engineers tuning LLM serving throughput follow scheduler internals like this via daily.dev._ ### What throughput improvement did vLLM show over naive Hugging Face serving in Anyscale's OPT-13B benchmark? vLLM reached 23 times the throughput of naive Hugging Face serving on the same A100 GPU in Anyscale's benchmark using OPT-13B, while static batching fell to around 81 tokens per second as output lengths varied. None of that gain came from a faster model; it came entirely from continuous batching in the scheduler rather than the model itself. _Comparing serving engine throughput numbers like these helps when picking an LLM inference stack on daily.dev._ ### What does max_num_batched_tokens control in vLLM's scheduler and how does it affect latency versus throughput? max_num_batched_tokens caps the total tokens the scheduler may put into one forward pass across every request in the step, not per request. Around 2048 tokens keeps inter-token latency tight but can leave the GPU under-fed, while around 16384 keeps the GPU saturated and raises throughput at the cost of every request waiting out a longer step; it also governs how chunked prefill splits work with pending decodes. _Developers tuning vLLM configs for latency versus throughput trade-offs can track details like this on daily.dev._ ## Similar posts on daily.dev - [Static vs. Dynamic vs. Continuous Batching in LLMs, clearly explained\!](https://daily.dev/posts/static-vs-dynamic-vs-continuous-batching-in-llms-clearly-explained--qbepnmqas) · Daily Dose of Data Science \| Avi Chawla \| Substack · 0 upvotes · 0 comments - [Continuous Batching vs. Static Batching in LLM Inference](https://daily.dev/posts/continuous-batching-vs-static-batching-in-llm-inference-xygdguiaz) · DigitalOcean Community · 0 upvotes · 0 comments - [Maximizing speed: How continuous batching unlocks unprecedented LLM throughput](https://daily.dev/posts/maximizing-speed-how-continuous-batching-unlocks-unprecedented-llm-throughput-30o1t1obl) · InfoWorld · 0 upvotes · 1 comments - [Continuous Batching Mechanics: The Scheduler Behind vLLM, TGI, and SGLang](https://daily.dev/posts/continuous-batching-mechanics-the-scheduler-behind-vllm-tgi-and-sglang-9x0j2tj3u) · DigitalOcean Community · 0 upvotes · 0 comments - [Continuous batching achieves high GPU utilization for large language models](https://daily.dev/posts/continuous-batching-achieves-high-gpu-utilization-for-large-language-models-z6b15ng2x) · Arpit Bhayani · 0 upvotes · 0 comments --- Tags: [#gpu](https://daily.dev/tags/gpu), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm), [#llmops](https://daily.dev/tags/llmops) [View this post on daily.dev](https://daily.dev/posts/continuous-batching-in-llms-yrzppfm2w) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Continuous Batching in LLMs","url":"https://daily.dev/posts/continuous-batching-in-llms-yrzppfm2w","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/continuous-batching-in-llms-yrzppfm2w"},"datePublished":"2026-08-13T20:06:42.277Z","dateModified":"2026-09-13T19:57:02.573Z","description":"Continuous batching is the scheduling technique used by default in serving engines like vLLM, SGLang, TGI, and TensorRT-LLM to handle LLM inference's...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/808aff842ba8bc0005d9c3a6eef97949?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/808aff842ba8bc0005d9c3a6eef97949?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Daily Dose of Data Science | Avi Chawla | Substack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Daily Dose of Data Science | Avi Chawla | Substack","logo":"https://media.daily.dev/image/upload/s--4IHQgTOw--/f_auto/v1710503712/logos/dailydoseofds","url":"https://daily.dev/sources/dailydoseofds"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/continuous-batching-in-llms-yrzppfm2w","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"gpu,ai-inference,vllm,llmops","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Daily Dose of Data Science | Avi Chawla | Substack","item":"https://daily.dev/sources/dailydoseofds"},{"@type":"ListItem","position":3,"name":"Continuous Batching in LLMs"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/continuous-batching-in-llms-yrzppfm2w#faq","mainEntity":[{"@type":"Question","name":"What is continuous batching in LLM inference and why does it improve throughput?","acceptedAnswer":{"@type":"Answer","text":"Continuous batching is a scheduling approach where the batch of requests is rebuilt at every forward pass instead of being fixed until all requests finish, so a finished request leaves immediately and a waiting request takes its slot rather than idling GPU capacity. It solves the problem where a fixed batch runs at the pace of its slowest member since output lengths vary and are unknown in advance. vLLM, SGLang, TGI, and TensorRT-LLM all use it by default. Engineers tuning LLM serving throughput follow scheduler internals like this via daily.dev."}},{"@type":"Question","name":"What throughput improvement did vLLM show over naive Hugging Face serving in Anyscale's OPT-13B benchmark?","acceptedAnswer":{"@type":"Answer","text":"vLLM reached 23 times the throughput of naive Hugging Face serving on the same A100 GPU in Anyscale's benchmark using OPT-13B, while static batching fell to around 81 tokens per second as output lengths varied. None of that gain came from a faster model; it came entirely from continuous batching in the scheduler rather than the model itself. Comparing serving engine throughput numbers like these helps when picking an LLM inference stack on daily.dev."}},{"@type":"Question","name":"What does max_num_batched_tokens control in vLLM's scheduler and how does it affect latency versus throughput?","acceptedAnswer":{"@type":"Answer","text":"max_num_batched_tokens caps the total tokens the scheduler may put into one forward pass across every request in the step, not per request. Around 2048 tokens keeps inter-token latency tight but can leave the GPU under-fed, while around 16384 keeps the GPU saturated and raises throughput at the cost of every request waiting out a longer step; it also governs how chunked prefill splits work with pending decodes. Developers tuning vLLM configs for latency versus throughput trade-offs can track details like this on daily.dev."}}]} ```