--- title: p50 vs p99 Latency: Why Median Benchmarks Mislead AI... description: Published inference benchmarks typically report median latency (p50), which hides the tail behavior that actually affects users. For a single interactive... canonical: https://daily.dev/posts/p50-vs-p99-latency-why-median-benchmarks-mislead-ai-agent-workloads-mn6r5fvtj twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: p50 vs p99 Latency: Why Median Benchmarks Mislead AI Agent Workloads | daily.dev og:description: Published inference benchmarks typically report median latency (p50), which hides the tail behavior that actually affects users. For a single interactive... og:url: https://daily.dev/posts/p50-vs-p99-latency-why-median-benchmarks-mislead-ai-agent-workloads-mn6r5fvtj og:image: https://api.daily.dev/og/posts/MN6r5fVtj.png og:image:alt: p50 vs p99 Latency: Why Median Benchmarks Mislead AI Agent Workloads og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # p50 vs p99 Latency: Why Median Benchmarks Mislead AI Agent Workloads **[DigitalOcean Community](https://daily.dev/sources/do_community)** · 40 min read · 1 upvotes · 0 comments ## Summary Published inference benchmarks typically report median latency (p50), which hides the tail behavior that actually affects users. For a single interactive request, p95 and p99 percentiles matter far more than the median. For AI agent workloads making sequential model calls, the correct metric is total task completion time — the sum across all calls — not any individual call's latency. Using the formula P = 1 − (1 − q)^n, a 10-call agent chain with a 1% per-call tail rate has a ~9.6% chance of hitting at least one tail event, and at a 5% (p95) threshold that probability crosses 50% around 14 calls. Real measurements against DigitalOcean Serverless Inference with 1,590 requests showed that Llama 4 Maverick beat GPT-OSS 120B on TTFT by 4x (300ms vs 1,202ms), but lost the 10-call agent task by 28% at median and 84% at p95 due to slower generation speed. One Maverick chain stalled for 20.4 minutes despite a normal-looking TTFT. The article provides a full reproducible Python benchmark harness, a seven-question checklist for critically reading any published benchmark, and guidance on which metric to use for interactive, agent, and batch workloads. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why did Llama 4 Maverick lose a 10-call agent task even though it had 4x faster time to first token than GPT-OSS 120B? Because total completion time per call, not time to first token, determines task duration in a sequential agent chain. Maverick's median TTFT was 300ms versus GPT-OSS 120B's 1,202ms, but Maverick generated only 21 tokens per second versus GPT-OSS 120B's 86, so across ten sequential calls Maverick finished the task 28% slower at the median and 84% slower at p95. _Teams choosing models for agent pipelines can track measured chain-latency comparisons like this one on daily.dev._ ### How much more likely is an agent chain to hit a tail latency event compared to a single request? A 20-call agent chain with a 1% per-call tail rate (p99 threshold) has roughly an 18.2% chance that at least one call hits the tail, using the formula 1 minus (1 minus q) to the power n. At a 5% per-call tail rate (p95 threshold), a 10-call chain reaches about 40.1%, and the risk crosses 50% around 14 calls. _Engineers sizing timeout and retry budgets for agent chains can dig into latency compounding math like this via daily.dev._ ### How much can inference speed vary across providers serving the identical LLM? Speed differences across providers hosting the same model can be enormous even though the model itself is identical: Artificial Analysis reported a 963% output-speed spread across providers for GLM-5.2 and a 4,149% spread for gpt-oss-120b, with Cerebras reaching 1,677.5 tokens per second at the top end. This means provider choice can matter more than model choice for latency. _Anyone comparing inference providers for the same model can weigh spread data like this through daily.dev._ ## Similar posts on daily.dev - [How to Benchmark LLM Inference Yourself: A Design Standard for Numbers You Can Trust](https://daily.dev/posts/how-to-benchmark-llm-inference-yourself-a-design-standard-for-numbers-you-can-trust-ezzqvl6s8) · DigitalOcean Community · 0 upvotes · 0 comments - [Metrics that Matter with Serverless Inference](https://daily.dev/posts/metrics-that-matter-with-serverless-inference-9nynkhj1z) · DigitalOcean Community · 0 upvotes · 0 comments - [Tail Control: The Counterintuitive Engineering of Reliable Agentic Workflows](https://daily.dev/posts/tail-control-the-counterintuitive-engineering-of-reliable-agentic-workflows-xl1si9fcr) · Towards Data Science · 1 upvotes · 0 comments - [Local vs Cloud AI Coding: Latency, Privacy & Performance Guide](https://daily.dev/posts/local-vs-cloud-ai-coding-latency-privacy-performance-guide-ce3vygmeo) · SitePoint · 0 upvotes · 0 comments - [The 200ms latency: A developer’s guide to real-time personalization](https://daily.dev/posts/the-200ms-latency-a-developer-s-guide-to-real-time-personalization-u7s5bwkwz) · InfoWorld · 0 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#digitalocean](https://daily.dev/tags/digitalocean), [#ai-inference](https://daily.dev/tags/ai-inference) [View this post on daily.dev](https://daily.dev/posts/p50-vs-p99-latency-why-median-benchmarks-mislead-ai-agent-workloads-mn6r5fvtj) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"p50 vs p99 Latency: Why Median Benchmarks Mislead AI Agent Workloads","url":"https://daily.dev/posts/p50-vs-p99-latency-why-median-benchmarks-mislead-ai-agent-workloads-mn6r5fvtj","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/p50-vs-p99-latency-why-median-benchmarks-mislead-ai-agent-workloads-mn6r5fvtj"},"datePublished":"2026-07-29T20:36:36.912Z","dateModified":"2026-09-13T20:32:07.770Z","description":"Published inference benchmarks typically report median latency (p50), which hides the tail behavior that actually affects users. For a single interactive...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/adb560f33713a6f15a527be03ce2462b?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/adb560f33713a6f15a527be03ce2462b?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"DigitalOcean Community","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"DigitalOcean Community","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/c1b9d07730e34ea388c39a498a753d6c","url":"https://daily.dev/sources/do_community"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/p50-vs-p99-latency-why-median-benchmarks-mislead-ai-agent-workloads-mn6r5fvtj","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,digitalocean,ai-inference","timeRequired":"PT40M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DigitalOcean Community","item":"https://daily.dev/sources/do_community"},{"@type":"ListItem","position":3,"name":"p50 vs p99 Latency: Why Median Benchmarks Mislead AI Agent Workloads"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/p50-vs-p99-latency-why-median-benchmarks-mislead-ai-agent-workloads-mn6r5fvtj#faq","mainEntity":[{"@type":"Question","name":"Why did Llama 4 Maverick lose a 10-call agent task even though it had 4x faster time to first token than GPT-OSS 120B?","acceptedAnswer":{"@type":"Answer","text":"Because total completion time per call, not time to first token, determines task duration in a sequential agent chain. Maverick's median TTFT was 300ms versus GPT-OSS 120B's 1,202ms, but Maverick generated only 21 tokens per second versus GPT-OSS 120B's 86, so across ten sequential calls Maverick finished the task 28% slower at the median and 84% slower at p95. Teams choosing models for agent pipelines can track measured chain-latency comparisons like this one on daily.dev."}},{"@type":"Question","name":"How much more likely is an agent chain to hit a tail latency event compared to a single request?","acceptedAnswer":{"@type":"Answer","text":"A 20-call agent chain with a 1% per-call tail rate (p99 threshold) has roughly an 18.2% chance that at least one call hits the tail, using the formula 1 minus (1 minus q) to the power n. At a 5% per-call tail rate (p95 threshold), a 10-call chain reaches about 40.1%, and the risk crosses 50% around 14 calls. Engineers sizing timeout and retry budgets for agent chains can dig into latency compounding math like this via daily.dev."}},{"@type":"Question","name":"How much can inference speed vary across providers serving the identical LLM?","acceptedAnswer":{"@type":"Answer","text":"Speed differences across providers hosting the same model can be enormous even though the model itself is identical: Artificial Analysis reported a 963% output-speed spread across providers for GLM-5.2 and a 4,149% spread for gpt-oss-120b, with Cerebras reaching 1,677.5 tokens per second at the top end. This means provider choice can matter more than model choice for latency. Anyone comparing inference providers for the same model can weigh spread data like this through daily.dev."}}]} ```