--- title: 5 Billion Records, 10 Terabytes, Four Weeks: A Real Data... description: Wix Engineering shares lessons from running large-scale data migrations, including a 5-billion-record, 10-terabyte migration completed in four weeks compared... canonical: https://daily.dev/posts/5-billion-records-10-terabytes-four-weeks-a-real-data-migration-playbook-finv6kwcs twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: 5 Billion Records, 10 Terabytes, Four Weeks: A Real Data Migration Playbook | daily.dev og:description: Wix Engineering shares lessons from running large-scale data migrations, including a 5-billion-record, 10-terabyte migration completed in four weeks compared... og:url: https://daily.dev/posts/5-billion-records-10-terabytes-four-weeks-a-real-data-migration-playbook-finv6kwcs og:image: https://api.daily.dev/og/posts/FINV6KWCs.png og:image:alt: 5 Billion Records, 10 Terabytes, Four Weeks: A Real Data Migration Playbook og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # 5 Billion Records, 10 Terabytes, Four Weeks: A Real Data Migration Playbook **[Wix Engineering](https://daily.dev/sources/wix_eng)** · 11 min read · 0 upvotes · 1 comments ## Summary Wix Engineering shares lessons from running large-scale data migrations, including a 5-billion-record, 10-terabyte migration completed in four weeks compared to past migrations that took months or years. Key practices include profiling source database quality before writing migration code, catching parity gaps between V1 and V2 APIs via E2E tests, LLM-assisted analysis, and compare-mode shadow traffic. It covers strategies for handling failed CDC messages (blocking retries, try-catch-skip, dead letter queues), calculating migration throughput against incoming change rates, using Kafka partitions and database sharding to scale speed, decoupling side effects to reduce blast radius, and treating full cleanup (deleting V1 code, pods, and databases) as part of the migration. It closes with speculation about supervised, agentic migrations handling the repetitive playbook steps in the future. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How should I handle a single bad record breaking a CDC-based migration pipeline? Three approaches exist depending on data trust level: blocking retries stop the pipeline until a human fixes the handler, ensuring nothing is silently skipped but sacrificing speed; try-catch-skip logs the failure with context and continues, letting you triage stragglers afterward; and a dead letter queue diverts failed messages to a separate topic so the main pipeline never stalls, offering the most robustness at the highest build cost. _Teams designing resilient CDC pipelines can find migration failure-handling patterns like these on daily.dev._ ### How many Kafka CDC partitions do I need to migrate a database with high write throughput? Partition count must be calculated before starting the migration by dividing required throughput by the expected per-partition processing rate, since adding partitions mid-migration is not possible. In one contacts migration handling roughly 15,000-50,000 RPM of writes, 350 partitions were needed to sustain 100,000 RPM of migration throughput. _Engineers sizing CDC migrations for scale can track throughput planning approaches like this on daily.dev._ ### Why would a large-scale eager data migration never finish even without any errors? If the migration processes historical records slower than new changes accumulate, it will never catch up regardless of how long it runs. In one case, 5 billion records needed migrating at a capped rate of 5,000 RPM to protect live traffic, but the database was accumulating roughly 15,000 RPM of new changes, meaning the migration had to either run fast enough to outrun the incoming change rate or take years. _Anyone estimating migration timelines against live write load can compare real throughput math like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@raunakpersonal** · 0 upvotes > Four weeks for 10 TB and 5 billion records makes validation strategy as important as transfer speed. The most interesting details would be how you handled dual writes, reconciliation, and rollback while keeping the source system live. ## Similar posts on daily.dev - [The Hi-Tech Data Convergence Migration Playbook](https://daily.dev/posts/the-hi-tech-data-convergence-migration-playbook-laxk1o9it) · SingleStore · 0 upvotes · 0 comments - [Migrating Data Ingestion Systems at Meta Scale](https://daily.dev/posts/migrating-data-ingestion-systems-at-meta-scale-tymkhmtiz) · Facebook Engineering · 4 upvotes · 1 comments --- Tags: [#career](https://daily.dev/tags/career), [#kafka](https://daily.dev/tags/kafka), [#change-data-capture](https://daily.dev/tags/change-data-capture) [View this post on daily.dev](https://daily.dev/posts/5-billion-records-10-terabytes-four-weeks-a-real-data-migration-playbook-finv6kwcs) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"5 Billion Records, 10 Terabytes, Four Weeks: A Real Data Migration Playbook","url":"https://daily.dev/posts/5-billion-records-10-terabytes-four-weeks-a-real-data-migration-playbook-finv6kwcs","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/5-billion-records-10-terabytes-four-weeks-a-real-data-migration-playbook-finv6kwcs"},"datePublished":"2026-09-02T07:28:06.867Z","dateModified":"2026-09-14T08:51:57.364Z","description":"Wix Engineering shares lessons from running large-scale data migrations, including a 5-billion-record, 10-terabyte migration completed in four weeks compared...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4817025bf589d6f73f85b708dec8b8e0?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4817025bf589d6f73f85b708dec8b8e0?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Wix Engineering","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Wix Engineering","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/5a14a3f3d2a14128bcb9a084fb2ebadf","url":"https://daily.dev/sources/wix_eng"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/5-billion-records-10-terabytes-four-weeks-a-real-data-migration-playbook-finv6kwcs","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"career,kafka,change-data-capture","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Wix Engineering","item":"https://daily.dev/sources/wix_eng"},{"@type":"ListItem","position":3,"name":"5 Billion Records, 10 Terabytes, Four Weeks: A Real Data Migration Playbook"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/5-billion-records-10-terabytes-four-weeks-a-real-data-migration-playbook-finv6kwcs","comment":[{"@type":"Comment","text":"Four weeks for 10 TB and 5 billion records makes validation strategy as important as transfer speed. The most interesting details would be how you handled dual writes, reconciliation, and rollback while keeping the source system live.","datePublished":"2026-09-02T08:15:55.945Z","url":"https://daily.dev/posts/FINV6KWCs#c-TcirpWKh4","author":{"@type":"Person","name":"Raunak Personal","url":"https://daily.dev/raunakpersonal","image":"https://media.daily.dev/image/upload/s--OBhKXyMB--/f_auto/v1773905110/avatars/avatar_fvBRceLNSvaKf8Z9ZAmaT?_a=BAMAMiiu0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/5-billion-records-10-terabytes-four-weeks-a-real-data-migration-playbook-finv6kwcs#faq","mainEntity":[{"@type":"Question","name":"How should I handle a single bad record breaking a CDC-based migration pipeline?","acceptedAnswer":{"@type":"Answer","text":"Three approaches exist depending on data trust level: blocking retries stop the pipeline until a human fixes the handler, ensuring nothing is silently skipped but sacrificing speed; try-catch-skip logs the failure with context and continues, letting you triage stragglers afterward; and a dead letter queue diverts failed messages to a separate topic so the main pipeline never stalls, offering the most robustness at the highest build cost. Teams designing resilient CDC pipelines can find migration failure-handling patterns like these on daily.dev."}},{"@type":"Question","name":"How many Kafka CDC partitions do I need to migrate a database with high write throughput?","acceptedAnswer":{"@type":"Answer","text":"Partition count must be calculated before starting the migration by dividing required throughput by the expected per-partition processing rate, since adding partitions mid-migration is not possible. In one contacts migration handling roughly 15,000-50,000 RPM of writes, 350 partitions were needed to sustain 100,000 RPM of migration throughput. Engineers sizing CDC migrations for scale can track throughput planning approaches like this on daily.dev."}},{"@type":"Question","name":"Why would a large-scale eager data migration never finish even without any errors?","acceptedAnswer":{"@type":"Answer","text":"If the migration processes historical records slower than new changes accumulate, it will never catch up regardless of how long it runs. In one case, 5 billion records needed migrating at a capped rate of 5,000 RPM to protect live traffic, but the database was accumulating roughly 15,000 RPM of new changes, meaning the migration had to either run fast enough to outrun the incoming change rate or take years. Anyone estimating migration timelines against live write load can compare real throughput math like this on daily.dev."}}]} ```