Netflix replaced Casspactor, their legacy Cassandra-to-Iceberg data movement engine handling ~1,200 jobs/day and ~3 PB of data, with a new layered architecture built on Apache Cassandra Analytics and a Move Data framework. The old system suffered from fragile multi-service metadata dependencies, inability to handle skewed partitions, monolithic design, and costly intermediate Iceberg tables. The new stack reads SSTable backups directly from S3, produces standard Spark DataFrames, supports a Connector Factory model for data-model-aware connectors, adds time travel, auto-sizing, and eliminates intermediate tables — saving millions in storage and compute costs. Migration was executed using three pillars: shadow-mode Validation (row-by-row comparison), Visibility (dashboards and alerting), and Safety (a Decider pattern in the Maestro orchestration platform that transparently fell back to Casspactor on failure), achieving a zero-impact transition for all downstream teams.

15m read timeFrom medium.com
Post cover image
Table of contents
BackgroundCasspactor: The Engine We OutgrewFragile Metadata DependenciesEvery Connector Inherited Casspactor’s LimitationsThe New Stack: A Layered ArchitectureThe Journey Towards a Safe MigrationGet Netflix Technology Blog’s stories in your inboxPillar 1: ValidationUncovering and Resolving DisparitiesPillar 2: VisibilityPillar 3: SafetyLeveraging Abstraction: The Decider PatternConclusionLooking ahead
45.1K Impressions