Comparing three ways to stream data into Apache Iceberg tables: Spark Structured Streaming, Apache Flink's Iceberg sink, and the Kafka Connect Iceberg Sink Connector. Each approach trades off latency against the small-file problem created by frequent commits. Flink offers the lowest latency with exactly-once semantics and CDC support but generates the most small files; Spark suits teams already on batch pipelines; Kafka Connect offers operational simplicity for existing Kafka shops. A production pattern pairs streaming ingestion with hourly compaction and daily snapshot expiration, and a table showing files-per-day at different commit intervals illustrates the trade-off. Recommends monitoring commit frequency, average file size, compaction lag, and end-to-end latency, and targeting 1-5 minute commits with 32-128MB files compacted into 256MB targets.
Table of contents
Table of ContentsThree Streaming ArchitecturesThe Streaming + Compaction CycleThe Latency vs. Maintenance Trade-offProduction Streaming ArchitectureChoosing the Right ApproachTry Dremio Cloud free for 30 daysQuestions this post answers
What's the difference between using Flink versus Spark Structured Streaming to write to Iceberg tables?
Flink commits at checkpoint intervals (often every 30 seconds) and provides exactly-once delivery guarantees plus native CDC support via changelog streams, but produces a high volume of small files. Spark Structured Streaming uses configurable trigger intervals (e.g., 60 seconds), fits teams already running Spark batch jobs, and produces moderately fewer small files at longer trigger intervals. daily.dev surfaces engineering comparisons like this for teams choosing a streaming ingestion engine for iceberg.
How often should I run compaction on a streaming Iceberg table?
A well-tuned pipeline commits data every 1-5 minutes producing files of 32-128 MB, then runs compaction every 30-60 minutes to consolidate those small files into 256 MB targets, with snapshot expiration run daily to clean up metadata. Skipping the compaction service is the most common architectural mistake and causes query performance to degrade within days. teams tuning iceberg compaction schedules can track patterns like this on daily.dev.
Does the Kafka Connect Iceberg Sink Connector support schema evolution?
Yes, the community-maintained Iceberg Sink Connector for Kafka Connect supports schema evolution sourced from Kafka's Schema Registry, along with automatic table creation and partition routing. It reads records from Kafka topics, buffers them in memory, and commits to Iceberg on configurable batch intervals, offering lower small-file impact than Spark or Flink since commits happen less frequently. developers evaluating managed Kafka-to-Iceberg pipelines can follow tooling updates like this on daily.dev.