--- title: How to Scale an Integration Pipeline Without Breaking... description: A practitioner's account of scaling an enterprise integration pipeline from 500 to 8,000 events per second without sacrificing correctness. Two invariants... canonical: https://daily.dev/posts/how-to-scale-an-integration-pipeline-without-breaking-correctness-ghschovtz twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How to Scale an Integration Pipeline Without Breaking Correctness | daily.dev og:description: A practitioner's account of scaling an enterprise integration pipeline from 500 to 8,000 events per second without sacrificing correctness. Two invariants... og:url: https://daily.dev/posts/how-to-scale-an-integration-pipeline-without-breaking-correctness-ghschovtz og:image: https://api.daily.dev/og/posts/GHSChOvTZ.png og:image:alt: How to Scale an Integration Pipeline Without Breaking Correctness og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How to Scale an Integration Pipeline Without Breaking Correctness **[Towards Data Science](https://daily.dev/sources/tds)** · 17 min read · 1 upvotes · 0 comments ## Summary A practitioner's account of scaling an enterprise integration pipeline from 500 to 8,000 events per second without sacrificing correctness. Two invariants anchor everything: version-based last-write-wins to prevent stale overwrites, and transactionally-coupled dedup logging to guarantee exactly-once semantics. Techniques covered include entity-keyed partitioning with adaptive sub-partitioning for hot entities, micro-batching (100 records or 50ms) with per-record savepoints, and three-tier backpressure (self-throttling consumers, producer rate limits, priority-based load shedding). A real incident walkthrough shows circuit breakers, backpressure, and shedding working together during an ERP outage, with a reconciliation report of 23,000 events, 22,987 auto-replayed, 13 in DLQ, zero data loss. Closes by tying the approach to academic literature on skewed key partitioning, effectively-once semantics, and reactive backpressure. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I handle out-of-order and duplicate event delivery in a distributed integration pipeline without breaking data correctness? Use a version number owned by the source system on every entity, and reject writes with a SQL condition like WHERE version < new_version so a stale update can never overwrite a newer one. Pair this with a dedup log written in the same database transaction as the business data, enforced by a primary-key constraint, so duplicate detection can never drift from what was actually committed. _Developers wiring up idempotent write paths can find more real-world pipeline patterns like this on daily.dev._ ### How much throughput improvement can micro-batching give a database-writing consumer versus processing one record at a time? Micro-batching can produce roughly a sixteen-fold throughput increase: one production pipeline went from about 500 events per second processed one at a time to about 8,000 events per second when accumulating batches of 100 records (or 50ms, whichever came first) and deduping and committing them together, because the gain comes from collapsing many network round-trips into one or two rather than from added CPU. _Anyone tuning batch sizes for a real workload can compare notes on similar production tradeoffs via daily.dev._ ### How do I debug a connection pool starvation issue that shows up as request timeouts instead of connection errors? Pool starvation shows up as rising tail latency with a flat error rate, not connection-refused errors, because requests sit queued in the pool waiting for a free connection rather than failing outright. This can happen after a downstream service splits into separate read and write hostnames without the connection pool config being updated to account for the split, halving effective capacity to each host. _Engineers chasing latency spikes with no matching errors can track similar pipeline failure patterns on daily.dev._ ## Similar posts on daily.dev - [Ensuring reliable OpenTelemetry ingestion at scale](https://daily.dev/posts/ensuring-reliable-opentelemetry-ingestion-at-scale-r8929yipq) · ClickHouse · 3 upvotes · 0 comments - [Scaling Java-Based Real-Time Systems: The Hidden Tradeoffs of Event-Driven Design](https://daily.dev/posts/scaling-java-based-real-time-systems-the-hidden-tradeoffs-of-event-driven-design-ef85inlwt) · InfoQ · 8 upvotes · 1 comments - [How to Collect Server Events Across Hundreds of Microservices](https://daily.dev/posts/how-to-collect-server-events-across-hundreds-of-microservices-orjlpldo1) · ITNEXT · 0 upvotes · 0 comments - [Why Batch ELT Breaks at Enterprise Scale](https://daily.dev/posts/why-batch-elt-breaks-at-enterprise-scale-2uoypymjg) · Confluent Blog · 0 upvotes · 0 comments - [From Kafka to Databend Cloud: Engineering a Trillion-Scale Agent Trace Ingestion Pipeline](https://daily.dev/posts/from-kafka-to-databend-cloud-engineering-a-trillion-scale-agent-trace-ingestion-pipeline-4qtnulddc) · Databend Blog · 1 upvotes · 0 comments --- Tags: [#java](https://daily.dev/tags/java), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#kafka](https://daily.dev/tags/kafka) [View this post on daily.dev](https://daily.dev/posts/how-to-scale-an-integration-pipeline-without-breaking-correctness-ghschovtz) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How to Scale an Integration Pipeline Without Breaking Correctness","url":"https://daily.dev/posts/how-to-scale-an-integration-pipeline-without-breaking-correctness-ghschovtz","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-to-scale-an-integration-pipeline-without-breaking-correctness-ghschovtz"},"datePublished":"2026-08-19T18:59:29.912Z","dateModified":"2026-09-14T07:55:24.832Z","description":"A practitioner's account of scaling an enterprise integration pipeline from 500 to 8,000 events per second without sacrificing correctness. Two invariants...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d1d6d9885506a843a2ccef5dd4969bce?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d1d6d9885506a843a2ccef5dd4969bce?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Towards Data Science","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Towards Data Science","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/tds","url":"https://daily.dev/sources/tds"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-to-scale-an-integration-pipeline-without-breaking-correctness-ghschovtz","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"java,distributed-systems,kafka","timeRequired":"PT17M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Towards Data Science","item":"https://daily.dev/sources/tds"},{"@type":"ListItem","position":3,"name":"How to Scale an Integration Pipeline Without Breaking Correctness"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-to-scale-an-integration-pipeline-without-breaking-correctness-ghschovtz#faq","mainEntity":[{"@type":"Question","name":"How can I handle out-of-order and duplicate event delivery in a distributed integration pipeline without breaking data correctness?","acceptedAnswer":{"@type":"Answer","text":"Use a version number owned by the source system on every entity, and reject writes with a SQL condition like WHERE version < new_version so a stale update can never overwrite a newer one. Pair this with a dedup log written in the same database transaction as the business data, enforced by a primary-key constraint, so duplicate detection can never drift from what was actually committed. Developers wiring up idempotent write paths can find more real-world pipeline patterns like this on daily.dev."}},{"@type":"Question","name":"How much throughput improvement can micro-batching give a database-writing consumer versus processing one record at a time?","acceptedAnswer":{"@type":"Answer","text":"Micro-batching can produce roughly a sixteen-fold throughput increase: one production pipeline went from about 500 events per second processed one at a time to about 8,000 events per second when accumulating batches of 100 records (or 50ms, whichever came first) and deduping and committing them together, because the gain comes from collapsing many network round-trips into one or two rather than from added CPU. Anyone tuning batch sizes for a real workload can compare notes on similar production tradeoffs via daily.dev."}},{"@type":"Question","name":"How do I debug a connection pool starvation issue that shows up as request timeouts instead of connection errors?","acceptedAnswer":{"@type":"Answer","text":"Pool starvation shows up as rising tail latency with a flat error rate, not connection-refused errors, because requests sit queued in the pool waiting for a free connection rather than failing outright. This can happen after a downstream service splits into separate read and write hostnames without the connection pool config being updated to account for the split, halving effective capacity to each host. Engineers chasing latency spikes with no matching errors can track similar pipeline failure patterns on daily.dev."}}]} ```