--- title: Day 9/30 Real-World DevOps Problems | daily.dev description: A scenario-based puzzle walks through a checkout latency regression after a fraud-check API call was added inside a database transaction. While p50 latency... canonical: https://daily.dev/posts/day-9-30-real-world-devops-problems-eugdisbkp twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Day 9/30 Real-World DevOps Problems | daily.dev og:description: A scenario-based puzzle walks through a checkout latency regression after a fraud-check API call was added inside a database transaction. While p50 latency... og:url: https://daily.dev/posts/day-9-30-real-world-devops-problems-eugdisbkp og:image: https://api.daily.dev/og/posts/EugDiSbKp.png og:image:alt: Day 9/30 Real-World DevOps Problems og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Day 9/30 Real-World DevOps Problems **[Bobby Iliev](https://daily.dev/sources/9adj6fgmy)** · [@bobbyiliev](https://daily.dev/bobbyiliev) · 1 min read · 19 upvotes · 12 comments ## Summary A scenario-based puzzle walks through a checkout latency regression after a fraud-check API call was added inside a database transaction. While p50 latency increased as expected, p99 latency ballooned from 300ms to 3.8s despite normal CPU, memory, error rate, and GC metrics. The puzzle asks readers to diagnose the cause among four multiple-choice options, with the underlying lesson being that holding a database connection open for an extra 400ms during a fixed-size connection pool (20 connections, 45 checkouts/sec) causes connection queuing that disproportionately impacts tail latency even though the external call itself and database queries remain fast. ## Content Yesterday's release added a call to an external fraud-check API (about 400 ms) to checkout. Checkout p50 went from 80 ms to 500 ms. You expected that. But checkout p99 at peak went from 300 ms to 3.8 s. CPU is 35%. Memory is 50%. Error rate is 0.1%. Database query p99 is 12 ms. The fraud check runs inside the database transaction. Each pod has a pool of 20 database connections and handles about 45 checkouts per second at peak. Traces show the fraud call itself stays under 500 ms at p99, and GC pause times have not changed. What is causing the p99? A) The fraud-check API is slow at peak; ask the vendor for a better latency SLA and a timeout B) The new code allocates more, so GC pauses got longer; tune the garbage collector settings C) Requests queue for a database connection, because checkout now holds one 400 ms longer D) The database is overloaded at peak and needs a read replica to take the checkout reads Where would you look first? ## Community discussion Top comments from developers on daily.dev. **@bobbyiliev** · 3 upvotes >  **@bo\_iliev21** · 2 upvotes > C) Connection pool saturation. Holding a DB connection while waiting for an external API call is a great way to destroy your p99. > > The database is fine. The connection pool has simply become a very exclusive waiting room. 😂 > > I'd check connection acquisition latency and pool utilization first. **@alexdo** · 2 upvotes > C bro **@kholnnaw** · 1 upvotes > when a flaky CI pipeline shows up because of external API rate limits, do you prefer adding exponential backoff retries in the jobs or gating merges until the service stabilizes **@killmonger99** · 1 upvotes > C) Requests queue for a database connection, because checkout now holds one 400 ms longer. > > Database connection pool saturation caused by extending the transaction across a slow external API call. ## Similar posts on daily.dev - [5 Things That Cause High Latency in Your APIs \(and How to Fix Them\)](https://daily.dev/posts/5-things-that-cause-high-latency-in-your-apis-and-how-to-fix-them--ygnrjvfij) · Nordic APIs · 0 upvotes · 0 comments - [Part 3: The Insight — Catching the Bottleneck Red-Handed](https://daily.dev/posts/part-3-the-insight-catching-the-bottleneck-red-handed-sumo06zk4) · ITNEXT · 2 upvotes · 0 comments - [The 200ms latency: A developer’s guide to real-time personalization](https://daily.dev/posts/the-200ms-latency-a-developer-s-guide-to-real-time-personalization-u7s5bwkwz) · InfoWorld · 0 upvotes · 0 comments - [Database Performance Health Check: Six Months Slow, Fixed in 75 Minutes](https://daily.dev/posts/database-performance-health-check-six-months-slow-fixed-in-75-minutes-kyepjbsdt) · SQL Authority · 9 upvotes · 1 comments - [The 13-Second Request](https://daily.dev/posts/the-13-second-request-sh8zcd0r2) · Medium · 0 upvotes · 0 comments --- Tags: [#devops](https://daily.dev/tags/devops), [#career](https://daily.dev/tags/career) [View this post on daily.dev](https://daily.dev/posts/day-9-30-real-world-devops-problems-eugdisbkp) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/day-9-30-real-world-devops-problems-eugdisbkp","headline":"Day 9/30 Real-World DevOps Problems","text":"A scenario-based puzzle walks through a checkout latency regression after a fraud-check API call was added inside a database transaction. While p50 latency increased as expected, p99 latency ballooned from 300ms to 3.8s despite normal CPU, memory, error rate, and GC metrics. The puzzle asks readers to diagnose the cause among four multiple-choice options, with the underlying lesson being that holding a database connection open for an extra 400ms during a fixed-size connection pool (20 connections, 45 checkouts/sec) causes connection queuing that disproportionately impacts tail latency even though the external call itself and database queries remain fast.","url":"https://daily.dev/posts/day-9-30-real-world-devops-problems-eugdisbkp","datePublished":"2026-10-16T11:04:56.320Z","dateModified":"2026-10-16T11:05:15.293Z","author":{"@type":"Person","name":"Bobby Iliev","url":"https://daily.dev/bobbyiliev","image":"https://avatars3.githubusercontent.com/u/21223421?v=4","description":"DevOps, DevEx, cloud & open source\n","worksFor":{"@type":"Organization","name":"Materialize","logo":"https://res.cloudinary.com/daily-now/image/upload/s--4mL2CrlK--/f_auto/v1725263959/companies/materialize"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":85530}},"image":"https://media.daily.dev/image/upload/s--joRZ5_4N--/f_auto,pg_1/v1792148696/posts/EugDiSbKp?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":19},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":12}],"comment":[{"@type":"Comment","text":"","datePublished":"2026-10-16T11:05:26.202Z","url":"https://daily.dev/posts/EugDiSbKp#c-V6qJsydTL","author":{"@type":"Person","name":"Bobby Iliev","url":"https://daily.dev/bobbyiliev","image":"https://avatars3.githubusercontent.com/u/21223421?v=4"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3}},{"@type":"Comment","text":"C) Connection pool saturation. Holding a DB connection while waiting for an external API call is a great way to destroy your p99.\nThe database is fine. The connection pool has simply become a very exclusive waiting room. 😂\nI’d check connection acquisition latency and pool utilization first.","datePublished":"2026-10-16T11:23:53.034Z","url":"https://daily.dev/posts/EugDiSbKp#c-H8LV0e5AH","author":{"@type":"Person","name":"Boyan Iliev","url":"https://daily.dev/bo_iliev21","image":"https://media.daily.dev/image/upload/s--6YaCMZ6E--/f_auto/v1785245730/avatars/avatar_dTbIq-v-M?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"C bro","datePublished":"2026-10-16T11:26:07.130Z","url":"https://daily.dev/posts/EugDiSbKp#c-EbKrJVmPL","author":{"@type":"Person","name":"AlexDO","url":"https://daily.dev/alexdo","image":"https://media.daily.dev/image/upload/s--WXW_Nwn2--/f_auto/v1789118002/avatars/avatar_2CAXJ6ar9?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"when a flaky CI pipeline shows up because of external API rate limits, do you prefer adding exponential backoff retries in the jobs or gating merges until the service stabilizes","datePublished":"2026-10-18T01:35:13.875Z","url":"https://daily.dev/posts/EugDiSbKp#c-ccfznNF4J","author":{"@type":"Person","name":"kholnnaw","url":"https://daily.dev/kholnnaw","image":"https://avatars.githubusercontent.com/u/342162242?v=4"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"C) Requests queue for a database connection, because checkout now holds one 400 ms longer.\nDatabase connection pool saturation caused by extending the transaction across a slow external API call.","datePublished":"2026-10-17T03:58:36.066Z","url":"https://daily.dev/posts/EugDiSbKp#c-reFb25mpw","author":{"@type":"Person","name":"Umang","url":"https://daily.dev/killmonger99","image":"https://media.daily.dev/image/upload/s--O0TOmw4y--/f_auto/v1715772965/public/noProfile"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/9adj6fgmy","name":"Bobby Iliev"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Bobby Iliev","item":"https://daily.dev/sources/9adj6fgmy"},{"@type":"ListItem","position":3,"name":"Day 9/30 Real-World DevOps Problems"}]} ```