A measured comparison of continuous batching vs static (gated) batching for LLM inference, run on a DigitalOcean H200 GPU Droplet with vLLM v0.24.0 and Llama 3.1 8B. The experiment shows continuous batching dramatically improves median TTFT (8x better than gated at p50) but shifts latency variance into the token stream as inter-token gaps. At 10 req/s, median worst inter-token gap hits 129.6 ms. Disabling chunked prefill widens the p99 worst-gap from 189.9 ms to 267.8 ms (1.41x), confirming chunked prefill's role in moderating stream stalls. Preemptions were zero throughout due to the H200's large KV cache headroom with an 8B model. The piece includes full harness code, raw JSON results, and a reproducibility checklist.
Table of contents
Test SetupTL;DRTable of Terms and ConceptsHow the two batching models actually workThe experimentMeasured results on a DigitalOcean H200 GPU DropletReading the measured results as a checklistWhen to use whichConclusionReferencesQuestions this post answers
What does disabling chunked prefill in vLLM do to p99 inter-token latency at 10 req/s?
Disabling chunked prefill in vLLM at 10 req/s widens the p99 worst inter-token gap from 189.9 ms to 267.8 ms — a 1.41x increase — without meaningfully changing time-to-first-token (249.4 ms vs 252.2 ms). The effect is isolated to the tail of stream stalls, not admission latency, confirming chunked prefill's role in smoothing decode interruptions caused by long-prompt prefill insertions. Engineers tuning vLLM for streaming workloads track chunked prefill behavior and p99 gap measurements on daily.dev.
How much worse is static batching TTFT compared to continuous batching in vLLM?
Static (gated, B=16) batching produces a median TTFT of 195.8 ms versus 24.1 ms for continuous batching at 10 req/s — roughly 8x worse at p50. At p99, gated TTFT is 637.2 ms versus 252.2 ms for continuous, about 2.5x worse. The gap comes from the fill-and-release cost: requests wait for the batch to form and for the slowest neighbor to finish before the next batch starts. Teams choosing between batching strategies for LLM APIs find benchmark comparisons like this on daily.dev.
When does vLLM trigger preemption (RECOMPUTE) during continuous batching and how can I observe it?
vLLM triggers RECOMPUTE preemption when the KV cache fills and the scheduler must evict an in-flight sequence, dropping its cache and rerunning its full prefill when capacity frees. This is observable via the vllm:num_preemptions_total Prometheus counter scraped from the /metrics endpoint. On an H200 with 141 GB running Llama 3.1 8B and a mixed short/medium/long trace, preemptions never fired — zero across all runs — due to the large KV headroom relative to the model size. Developers monitoring vLLM in production watch preemption counters and tail latency signals on daily.dev.
246 Impressions1 Comment