--- title: Request Queuing and Latency Under Burst Traffic: What... description: A 60-second traffic burst can leave an LLM inference service degraded for several minutes after traffic returns to normal. This tutorial shows how to load-test... canonical: https://daily.dev/posts/request-queuing-and-latency-under-burst-traffic-what-happens-when-your-model-gets-overloaded--tp4jwaeg4 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Request Queuing and Latency Under Burst Traffic: What Happens When Your Model Gets Overloaded? | daily.dev og:description: A 60-second traffic burst can leave an LLM inference service degraded for several minutes after traffic returns to normal. This tutorial shows how to load-test... og:url: https://daily.dev/posts/request-queuing-and-latency-under-burst-traffic-what-happens-when-your-model-gets-overloaded--tp4jwaeg4 og:image: https://api.daily.dev/og/posts/Tp4jWAEg4.png og:image:alt: Request Queuing and Latency Under Burst Traffic: What Happens When Your Model Gets Overloaded? og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Request Queuing and Latency Under Burst Traffic: What Happens When Your Model Gets Overloaded? **[DigitalOcean Community](https://daily.dev/sources/do_community)** · 15 min read · 0 upvotes · 0 comments ## Summary A 60-second traffic burst can leave an LLM inference service degraded for several minutes after traffic returns to normal. This tutorial shows how to load-test LLM inference at 2x, 5x, and 10x burst multipliers on DigitalOcean Serverless Inference, Dedicated Inference, or self-managed engines like vLLM on GPU Droplets or DOKS. It defines capacity as 'goodput', the requests per second that finish before the application's deadline, rather than raw requests per second. A worked example shows a service handling 10 requests/sec that admits 20/sec for 60 seconds needs about 5 extra minutes to drain the backlog once arrivals return to 8/sec. The guide also covers measuring queue wait with vLLM metrics like vllm:num_requests_waiting, and timing autoscaling until new capacity actually serves traffic. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is 'goodput' in LLM inference benchmarking and how is it different from requests per second? Goodput is the number of requests per second that complete successfully within a per-request deadline, not just the number processed. For example, if a service completes 60 requests per second but only 45 finish before the application's deadline, goodput is 45 per second. A service can look fast on raw throughput while actually serving very few users on time. _Anyone sizing LLM inference capacity can compare benchmarking approaches like this on daily.dev._ ### How long does it take for an LLM inference service to recover after a traffic burst ends? Recovery time depends on the gap between admitted arrival rate and effective processing capacity after the burst ends. In one worked example, a service with 10 requests/sec capacity that admits 20 requests/sec for 60 seconds builds a backlog of 600 requests; once arrivals drop back to 8/sec, only 2/sec of spare capacity remains to drain it, costing roughly 5 extra minutes of elevated latency, five times the burst duration. _Teams planning for traffic spikes can track burst-recovery math like this on daily.dev._ ### Which vLLM Prometheus metrics show queueing and latency inside the inference engine? vLLM exposes vllm:request_queue_time_seconds for time waiting before scheduling, vllm:time_to_first_token_seconds for engine-side TTFT, vllm:e2e_request_latency_seconds for end-to-end latency, vllm:num_requests_waiting for requests currently queued, and vllm:num_requests_running for requests currently in a batch. Metric names and histogram buckets can change between releases, so confirm them against the deployed version. _Engineers instrumenting vLLM deployments can follow metric changes like these on daily.dev._ ## Similar posts on daily.dev - [Autoscaling endpoints for LLM inference](https://daily.dev/posts/autoscaling-endpoints-for-llm-inference-ddy21ncoc) · Together AI · 0 upvotes · 0 comments - [The LLM Inference Trilemma: Throughput, Latency, Cost](https://daily.dev/posts/the-llm-inference-trilemma-throughput-latency-cost-ws3duzsii) · DigitalOcean · 0 upvotes · 0 comments - [Reduce LLM benchmarking costs with oversaturation detection](https://daily.dev/posts/reduce-llm-benchmarking-costs-with-oversaturation-detection-odyiu4kzs) · Red Hat Developer · 2 upvotes · 0 comments - [Equal Cost Is a Lie: The Strange Physics of GPU Traffic](https://daily.dev/posts/equal-cost-is-a-lie-the-strange-physics-of-gpu-traffic-9juoea82v) · Medium · 2 upvotes · 0 comments - [How to Benchmark LLM Inference Yourself: A Design Standard for Numbers You Can Trust](https://daily.dev/posts/how-to-benchmark-llm-inference-yourself-a-design-standard-for-numbers-you-can-trust-ezzqvl6s8) · DigitalOcean Community · 0 upvotes · 0 comments --- Tags: [#ai-inference](https://daily.dev/tags/ai-inference), [#digitalocean](https://daily.dev/tags/digitalocean), [#vllm](https://daily.dev/tags/vllm), [#load-testing](https://daily.dev/tags/load-testing) [View this post on daily.dev](https://daily.dev/posts/request-queuing-and-latency-under-burst-traffic-what-happens-when-your-model-gets-overloaded--tp4jwaeg4) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Request Queuing and Latency Under Burst Traffic: What Happens When Your Model Gets Overloaded?","url":"https://daily.dev/posts/request-queuing-and-latency-under-burst-traffic-what-happens-when-your-model-gets-overloaded--tp4jwaeg4","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/request-queuing-and-latency-under-burst-traffic-what-happens-when-your-model-gets-overloaded--tp4jwaeg4"},"datePublished":"2026-10-30T21:43:36.288Z","dateModified":"2026-10-30T21:44:28.628Z","description":"A 60-second traffic burst can leave an LLM inference service degraded for several minutes after traffic returns to normal. This tutorial shows how to load-test...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ddb4b57e4a6360db4437e2cf70e5c755?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ddb4b57e4a6360db4437e2cf70e5c755?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"DigitalOcean Community","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"DigitalOcean Community","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/c1b9d07730e34ea388c39a498a753d6c","url":"https://daily.dev/sources/do_community"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/request-queuing-and-latency-under-burst-traffic-what-happens-when-your-model-gets-overloaded--tp4jwaeg4","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-inference,digitalocean,vllm,load-testing","timeRequired":"PT15M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DigitalOcean Community","item":"https://daily.dev/sources/do_community"},{"@type":"ListItem","position":3,"name":"Request Queuing and Latency Under Burst Traffic: What Happens When Your Model Gets Overloaded?"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/request-queuing-and-latency-under-burst-traffic-what-happens-when-your-model-gets-overloaded--tp4jwaeg4#faq","mainEntity":[{"@type":"Question","name":"What is 'goodput' in LLM inference benchmarking and how is it different from requests per second?","acceptedAnswer":{"@type":"Answer","text":"Goodput is the number of requests per second that complete successfully within a per-request deadline, not just the number processed. For example, if a service completes 60 requests per second but only 45 finish before the application's deadline, goodput is 45 per second. A service can look fast on raw throughput while actually serving very few users on time. Anyone sizing LLM inference capacity can compare benchmarking approaches like this on daily.dev."}},{"@type":"Question","name":"How long does it take for an LLM inference service to recover after a traffic burst ends?","acceptedAnswer":{"@type":"Answer","text":"Recovery time depends on the gap between admitted arrival rate and effective processing capacity after the burst ends. In one worked example, a service with 10 requests/sec capacity that admits 20 requests/sec for 60 seconds builds a backlog of 600 requests; once arrivals drop back to 8/sec, only 2/sec of spare capacity remains to drain it, costing roughly 5 extra minutes of elevated latency, five times the burst duration. Teams planning for traffic spikes can track burst-recovery math like this on daily.dev."}},{"@type":"Question","name":"Which vLLM Prometheus metrics show queueing and latency inside the inference engine?","acceptedAnswer":{"@type":"Answer","text":"vLLM exposes vllm:request_queue_time_seconds for time waiting before scheduling, vllm:time_to_first_token_seconds for engine-side TTFT, vllm:e2e_request_latency_seconds for end-to-end latency, vllm:num_requests_waiting for requests currently queued, and vllm:num_requests_running for requests currently in a batch. Metric names and histogram buckets can change between releases, so confirm them against the deployed version. Engineers instrumenting vLLM deployments can follow metric changes like these on daily.dev."}}]} ```