--- title: The GPU Is Not Your Time-to-First-Token Bottleneck description: Time-to-first-token (TTFT) latency in LLM serving is usually not caused by GPU compute speed. A 2026 study on disaggregated LLM serving found prefill compute... canonical: https://daily.dev/posts/the-gpu-is-not-your-time-to-first-token-bottleneck-gjivhoj0l twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The GPU Is Not Your Time-to-First-Token Bottleneck | daily.dev og:description: Time-to-first-token (TTFT) latency in LLM serving is usually not caused by GPU compute speed. A 2026 study on disaggregated LLM serving found prefill compute... og:url: https://daily.dev/posts/the-gpu-is-not-your-time-to-first-token-bottleneck-gjivhoj0l og:image: https://api.daily.dev/og/posts/GjiVHoJ0l.png og:image:alt: The GPU Is Not Your Time-to-First-Token Bottleneck og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The GPU Is Not Your Time-to-First-Token Bottleneck **[DigitalOcean Community](https://daily.dev/sources/do_community)** · 15 min read · 1 upvotes · 0 comments ## Summary Time-to-first-token (TTFT) latency in LLM serving is usually not caused by GPU compute speed. A 2026 study on disaggregated LLM serving found prefill compute accounts for only 2-23% of P95 TTFT, while queuing and KV-cache transfer account for 77-98%. Two architectural decisions dominate: provisioning (scale-to-zero vs keep-warm, which sets cold-start cost of 30-90+ seconds) and scheduling (aggregated vs disaggregated serving, trading TTFT against TPOT smoothness). A smarter scheduler that let decode nodes absorb prefill work cut P95 TTFT by up to 81% in that study, four times more than a GPU upgrade could achieve. Faster GPUs only genuinely help in the very-long-context regime, such as million-token prompts where prefill compute becomes the dominant cost. The piece includes two experiment designs developers can run on DigitalOcean Serverless Inference and DOKS to measure their own bottlenecks. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why doesn't upgrading to a faster GPU fix slow time-to-first-token in LLM serving? Because in most production setups prefill compute, the part a faster GPU accelerates, accounts for only 2 to 23 percent of P95 time-to-first-token, while queuing and KV-cache transfer account for 77 to 98 percent. A 2P2D A100 cluster study found prefill nodes ran at 75 to 84 percent compute utilization, yet the tail latency lived in queuing and network transfer, not compute. _Anyone chasing first-token latency can dig into these TTFT breakdowns and inference tuning discussions on daily.dev._ ### What causes long cold-start delays in serverless LLM inference? Cold starts stem from container image pulls, model weight loading into VRAM, and CUDA context and graph capture, and they commonly total 30 to 90 seconds, sometimes several minutes with an uncached large image. This is roughly 1000x slower than a warm token generation step of about 30 milliseconds, and production traces show over 40 percent of functions saw cold-start rates above 25 percent within a 5-minute keep-alive window. _Teams weighing scale-to-zero against keep-warm replicas can track this kind of inference architecture guidance on daily.dev._ ### What is the trade-off between aggregated and disaggregated LLM serving? Aggregated serving collocates prefill and decode on the same GPU with no network transfer, protecting time-to-first-token but risking interference where a long prefill stalls queued decode requests, hurting stream smoothness (TPOT). Disaggregated serving runs prefill and decode on separate GPU pools, protecting TPOT but adding KV-cache transfer time to TTFT, which one AMD MI300X evaluation measured as roughly 1.9x higher TTFT on average. _Engineers deciding between serving architectures can follow real-world TTFT versus TPOT trade-off analysis on daily.dev._ ## Similar posts on daily.dev - [Prefill Is Compute-Bound. Decode Is Memory-Bound. Why Your GPU Shouldn’t Do Both.](https://daily.dev/posts/prefill-is-compute-bound-decode-is-memory-bound-why-your-gpu-shouldn-t-do-both--eaagsfeyw) · Towards Data Science · 0 upvotes · 0 comments - [The Hidden Bottlenecks in LLM Inference and How to Fix Them](https://daily.dev/posts/the-hidden-bottlenecks-in-llm-inference-and-how-to-fix-them-q1fibdwak) · DigitalOcean Community · 0 upvotes · 0 comments - [How I doubled my GPU efficiency without buying a single new card](https://daily.dev/posts/how-i-doubled-my-gpu-efficiency-without-buying-a-single-new-card-e57gspube) · InfoWorld · 0 upvotes · 0 comments - [The Inference Bottleneck: Architecting Kubernetes Autoscaling for Production LLMs](https://daily.dev/posts/the-inference-bottleneck-architecting-kubernetes-autoscaling-for-production-llms-3igbdeoqq) · Container Journal · 1 upvotes · 0 comments - [How to Diagnose and Fix AI Inference Latency on Kubernetes](https://daily.dev/posts/how-to-diagnose-and-fix-ai-inference-latency-on-kubernetes-flymwvtow) · freeCodeCamp · 1 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#gpu](https://daily.dev/tags/gpu), [#digitalocean](https://daily.dev/tags/digitalocean), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/the-gpu-is-not-your-time-to-first-token-bottleneck-gjivhoj0l) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"The GPU Is Not Your Time-to-First-Token Bottleneck","url":"https://daily.dev/posts/the-gpu-is-not-your-time-to-first-token-bottleneck-gjivhoj0l","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/the-gpu-is-not-your-time-to-first-token-bottleneck-gjivhoj0l"},"datePublished":"2026-08-19T01:57:27.487Z","dateModified":"2026-09-13T20:34:35.567Z","description":"Time-to-first-token (TTFT) latency in LLM serving is usually not caused by GPU compute speed. A 2026 study on disaggregated LLM serving found prefill compute...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c6bb65789b5513621a96a28f128f3a2a?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c6bb65789b5513621a96a28f128f3a2a?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"DigitalOcean Community","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"DigitalOcean Community","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/c1b9d07730e34ea388c39a498a753d6c","url":"https://daily.dev/sources/do_community"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/the-gpu-is-not-your-time-to-first-token-bottleneck-gjivhoj0l","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,gpu,digitalocean,ai-inference,vllm","timeRequired":"PT15M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DigitalOcean Community","item":"https://daily.dev/sources/do_community"},{"@type":"ListItem","position":3,"name":"The GPU Is Not Your Time-to-First-Token Bottleneck"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/the-gpu-is-not-your-time-to-first-token-bottleneck-gjivhoj0l#faq","mainEntity":[{"@type":"Question","name":"Why doesn't upgrading to a faster GPU fix slow time-to-first-token in LLM serving?","acceptedAnswer":{"@type":"Answer","text":"Because in most production setups prefill compute, the part a faster GPU accelerates, accounts for only 2 to 23 percent of P95 time-to-first-token, while queuing and KV-cache transfer account for 77 to 98 percent. A 2P2D A100 cluster study found prefill nodes ran at 75 to 84 percent compute utilization, yet the tail latency lived in queuing and network transfer, not compute. Anyone chasing first-token latency can dig into these TTFT breakdowns and inference tuning discussions on daily.dev."}},{"@type":"Question","name":"What causes long cold-start delays in serverless LLM inference?","acceptedAnswer":{"@type":"Answer","text":"Cold starts stem from container image pulls, model weight loading into VRAM, and CUDA context and graph capture, and they commonly total 30 to 90 seconds, sometimes several minutes with an uncached large image. This is roughly 1000x slower than a warm token generation step of about 30 milliseconds, and production traces show over 40 percent of functions saw cold-start rates above 25 percent within a 5-minute keep-alive window. Teams weighing scale-to-zero against keep-warm replicas can track this kind of inference architecture guidance on daily.dev."}},{"@type":"Question","name":"What is the trade-off between aggregated and disaggregated LLM serving?","acceptedAnswer":{"@type":"Answer","text":"Aggregated serving collocates prefill and decode on the same GPU with no network transfer, protecting time-to-first-token but risking interference where a long prefill stalls queued decode requests, hurting stream smoothness (TPOT). Disaggregated serving runs prefill and decode on separate GPU pools, protecting TPOT but adding KV-cache transfer time to TTFT, which one AMD MI300X evaluation measured as roughly 1.9x higher TTFT on average. Engineers deciding between serving architectures can follow real-world TTFT versus TPOT trade-off analysis on daily.dev."}}]} ```