--- title: Our ingest p50 said 66 minutes. Nothing in the system... description: An engineer debugging a misleading p50 latency metric for a financial news ingest pipeline found the number blended two distinct populations: a fast collector... canonical: https://daily.dev/posts/our-ingest-p50-said-66-minutes-nothing-in-the-system-actually-took-66-minutes--azers10yc twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Our ingest p50 said 66 minutes. Nothing in the system actually took 66 minutes. | daily.dev og:description: An engineer debugging a misleading p50 latency metric for a financial news ingest pipeline found the number blended two distinct populations: a fast collector... og:url: https://daily.dev/posts/our-ingest-p50-said-66-minutes-nothing-in-the-system-actually-took-66-minutes--azers10yc og:image: https://api.daily.dev/og/posts/AZerS10YC.png og:image:alt: Our ingest p50 said 66 minutes. Nothing in the system actually took 66 minutes. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Our ingest p50 said 66 minutes. Nothing in the system actually took 66 minutes. **[AlphAI](https://daily.dev/sources/alphai)** · [@mikhailmakeev](https://daily.dev/mikhailmakeev) · 2 min read · 3 upvotes · 2 comments ## Summary An engineer debugging a misleading p50 latency metric for a financial news ingest pipeline found the number blended two distinct populations: a fast collector channel (30 min median) and a slow Google News rotation channel bound by arithmetic (361 min median, walking 2,113 tickers across 10 feeds per cycle). Along the way they also found a polling loop bug where sleeping the full interval instead of interval-minus-elapsed stretched a 5-minute cycle to 9-12 minutes, and discovered that default OpenTelemetry histogram buckets cap at 10,000 seconds, silently swallowing the slow channel's ~19 hour p90 into an overflow bucket that looked healthy. The fix was splitting the two channels into separate containers with separate poll intervals and metric labels, plus a reminder to check what a timestamp actually means on each ingest path before comparing percentiles. ## Content We pull financial news from several collectors into one table, and I measured the delay from an article's publish timestamp to the row being queryable. Median 66 minutes. A competitor doing the same job measured 4, so I went looking for the slow part. There was no slow part. There were two populations in one number. Split by collector channel, 68% of rows land at a median of 30 minutes and 32% at a median of 361. The slow third is a Google News rotation that walks 2,113 tickers, 10 feeds per cycle. That is a 42-hour round trip by arithmetic, not by a bug: an article gets found when its ticker's turn comes up. Nothing there can be tuned into the fast channel's numbers, because one is a coverage tool and the other is a latency tool. Two smaller things fell out of the same look. The loop slept the full interval after the body instead of interval minus elapsed. With a 5 to 7 minute body and a 300 second setting, real cycles ran 9 to 12 minutes. Fixed delay where I had assumed fixed rate, so a fresh batch waited about 6 minutes just to be noticed. Then, adding a per-channel latency histogram, the default OpenTelemetry buckets stop at 10,000 seconds. My p90 on the slow channel is near 19 hours, so everything interesting would have landed in the overflow bucket and the panel would have looked healthy. The two channels now run as separate containers off one image, each with its own poll interval and its own metric label. The fast cycle is exactly 5 minutes. What I would carry elsewhere: before optimizing a percentile, check that it describes one population, and check what your timestamp means on each branch. Ours was the publisher's pubDate on one path and the aggregator's "we saw it" time on the other, so the two medians were never comparable to begin with. The feed behind those numbers has a free key at 20 requests a minute if you want to look at the timestamps yourself: https://alphai.io/developers ## Community discussion Top comments from developers on daily.dev. **@agustinbarrientos** · 2 upvotes > I'd alert on the overflow-bucket share before trusting any latency percentile from that dashboard ## Similar posts on daily.dev - [p50 vs p99 Latency: Why Median Benchmarks Mislead AI Agent Workloads](https://daily.dev/posts/p50-vs-p99-latency-why-median-benchmarks-mislead-ai-agent-workloads-mn6r5fvtj) · DigitalOcean Community · 1 upvotes · 0 comments - [The Hidden Problem with Kafka Lag Monitoring](https://daily.dev/posts/the-hidden-problem-with-kafka-lag-monitoring-ktemrnxfn) · SoftwareMill · 0 upvotes · 0 comments - [Create histogram metrics at ingest with OpenPipeline](https://daily.dev/posts/create-histogram-metrics-at-ingest-with-openpipeline-imvvjymcj) · Dynatrace · 0 upvotes · 0 comments - [The 200ms latency: A developer’s guide to real-time personalization](https://daily.dev/posts/the-200ms-latency-a-developer-s-guide-to-real-time-personalization-u7s5bwkwz) · InfoWorld · 0 upvotes · 0 comments - [The 3.5-Second Lie: How a Blind Spot in My Telemetry Hid Two Production Bugs](https://daily.dev/posts/the-3-5-second-lie-how-a-blind-spot-in-my-telemetry-hid-two-production-bugs-pzo1aofwi) · Medium · 0 upvotes · 0 comments --- Tags: [#observability](https://daily.dev/tags/observability), [#data-engineering](https://daily.dev/tags/data-engineering), [#opentelemetry](https://daily.dev/tags/opentelemetry) [View this post on daily.dev](https://daily.dev/posts/our-ingest-p50-said-66-minutes-nothing-in-the-system-actually-took-66-minutes--azers10yc) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/our-ingest-p50-said-66-minutes-nothing-in-the-system-actually-took-66-minutes--azers10yc","headline":"Our ingest p50 said 66 minutes. Nothing in the system actually took 66 minutes.","text":"An engineer debugging a misleading p50 latency metric for a financial news ingest pipeline found the number blended two distinct populations: a fast collector channel (30 min median) and a slow Google News rotation channel bound by arithmetic (361 min median, walking 2,113 tickers across 10 feeds per cycle). Along the way they also found a polling loop bug where sleeping the full interval instead of interval-minus-elapsed stretched a 5-minute cycle to 9-12 minutes, and discovered that default OpenTelemetry histogram buckets cap at 10,000 seconds, silently swallowing the slow channel's ~19 hour p90 into an overflow bucket that looked healthy. The fix was splitting the two channels into separate containers with separate poll intervals and metric labels, plus a reminder to check what a timestamp actually means on each ingest path before comparing percentiles.","url":"https://daily.dev/posts/our-ingest-p50-said-66-minutes-nothing-in-the-system-actually-took-66-minutes--azers10yc","datePublished":"2026-08-18T14:21:26.609Z","dateModified":"2026-09-13T20:55:49.185Z","author":{"@type":"Person","name":"Mikhail Makeev","url":"https://daily.dev/mikhailmakeev","image":"https://lh3.googleusercontent.com/a/ACg8ocIjcvDU1gmJ9-KJDeTYqY2rMdl5QxcWYtLTGSXrj8ubF4nwar_w=s96-c","description":"Owner of alphai.io","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":3750}},"image":"https://media.daily.dev/image/upload/s--HYZxJzA0--/f_auto/v1789277221/posts/AZerS10YC?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":2}],"comment":[{"@type":"Comment","text":"I’d alert on the overflow-bucket share before trusting any latency percentile from that dashboard","datePublished":"2026-08-18T20:28:59.822Z","url":"https://daily.dev/posts/AZerS10YC#c-JXWVYKUQ6","author":{"@type":"Person","name":"Agustin Barrientos","url":"https://daily.dev/agustinbarrientos","image":"https://media.daily.dev/image/upload/s--5ayxQnqn--/f_auto/v1788281802/avatars/avatar_wQYYVe5Tbj0NJ7C7qPoa8?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/squads/alphai","name":"AlphAI"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AlphAI","item":"https://daily.dev/squads/alphai"},{"@type":"ListItem","position":3,"name":"Our ingest p50 said 66 minutes. Nothing in the system actually took 66 minutes."}]} ```