Netflix TechBlog
Read post

The Evolution of Cassandra Data Movement at Netflix

Netflix replaced Casspactor, their legacy Cassandra-to-Iceberg data movement engine handling ~1,200 jobs/day and ~3 PB of data, with a new layered architecture built on Apache Cassandra Analytics and a Move Data framework. The old system suffered from fragile multi-service metadata dependencies, inability to handle skewed partitions, monolithic design, and costly intermediate Iceberg tables. The new stack reads SSTable backups directly from S3, produces standard Spark DataFrames, supports a Connector Factory model for data-model-aware connectors, adds time travel, auto-sizing, and eliminates intermediate tables — saving millions in storage and compute costs. Migration was executed using three pillars: shadow-mode Validation (row-by-row comparison), Visibility (dashboards and alerting), and Safety (a Decider pattern in the Maestro orchestration platform that transparently fell back to Casspactor on failure), achieving a zero-impact transition for all downstream teams.

    #backend#apache-spark#apache-cassandra#apache-iceberg
Jun 20•15m read time•From medium.com
Post cover image
Table of contents
BackgroundCasspactor: The Engine We OutgrewFragile Metadata DependenciesEvery Connector Inherited Casspactor’s LimitationsThe New Stack: A Layered ArchitectureThe Journey Towards a Safe MigrationGet Netflix Technology Blog’s stories in your inboxPillar 1: ValidationUncovering and Resolving DisparitiesPillar 2: VisibilityPillar 3: SafetyLeveraging Abstraction: The Decider PatternConclusionLooking ahead
45.1K Impressions
Netflix TechBlog's image
Netflix TechBlog

The Netflix Tech Blog provides insights into the engineering practices, architecture, and innovation...

597 Followers

•

1.2K Upvotes

Would you recommend this post?

Copy link
WhatsApp
Facebook
X
New Squad
  • © 2026 Daily Dev Ltd.
  • Guidelines
  • Explore
  • Tags
  • Sources
  • Squads
  • Leaderboard