This post discusses the trade-offs between prioritizing data availability over durability in Apache Kafka, illustrating how unclean leader election can help reduce service downtime at the cost of potential data loss. It provides insights into mitigating data loss by configuring replication factors, selecting appropriate replication strategies, and understanding the balance between preserving data and ensuring service continuity. Key considerations for Kafka configuration and the process of estimating data loss are also highlighted.
Table of contents
When keeping data perfect isn’t perfect: prioritizing availability in Apache KafkaWhy does data loss happen?Preventing offline partitions in KafkaKafka’s unclean leader electionThe Achilles HeelAvailability vs. durabilityEstimating data loss (after the fact)Is durability all that matters?23 Impressions