--- title: Writing a Kafka Stream to Delta Lake with Spark... description: A tutorial demonstrates how to ingest a Kafka stream into a Delta Lake table using Spark Structured Streaming, covering both periodic (availableNow trigger)... canonical: https://daily.dev/posts/writing-a-kafka-stream-to-delta-lake-with-spark-structured-streaming-my2gpwp39 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Writing a Kafka Stream to Delta Lake with Spark Structured Streaming | daily.dev og:description: A tutorial demonstrates how to ingest a Kafka stream into a Delta Lake table using Spark Structured Streaming, covering both periodic (availableNow trigger)... og:url: https://daily.dev/posts/writing-a-kafka-stream-to-delta-lake-with-spark-structured-streaming-my2gpwp39 og:image: https://api.daily.dev/og/posts/My2gpwp39.png og:image:alt: Writing a Kafka Stream to Delta Lake with Spark Structured Streaming og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Writing a Kafka Stream to Delta Lake with Spark Structured Streaming **[Delta Lake Blog](https://daily.dev/sources/delta-lake)** · 6 min read · 0 upvotes · 0 comments ## Summary A tutorial demonstrates how to ingest a Kafka stream into a Delta Lake table using Spark Structured Streaming, covering both periodic (availableNow trigger) and continuous processing approaches. It explains schema definition, JSON parsing, and checkpointing for incremental updates. The post also addresses the small file problem that streaming ingestion can create, especially with high-cardinality partition keys, and shows how Delta Lake's OPTIMIZE command compacts small files. It closes with a list of Delta Lake advantages for Kafka workloads: schema enforcement, schema evolution, concurrency control, DML support, and a broad connector ecosystem. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I avoid the small file problem when streaming Kafka data into a Delta Lake table? Design the ingestion pipeline to minimize small file creation, then periodically run Delta Lake's OPTIMIZE command to compact them. Streaming with low latency or partitioning by a high-cardinality column like medical_code tends to produce many small files, while partitioning by something like ingestion_date avoids the issue. Predicates let you compact only a subset, such as the previous day's ingested files, keeping the operation lightweight. _See how daily.dev surfaces practical Delta Lake and Spark streaming techniques for engineers managing pipeline performance._ ### Should I use availableNow trigger or continuous processing for streaming Kafka data into Delta Lake with Spark? Use availableNow for periodic, incremental processing when latency requirements are relaxed, since it avoids the cost of running a cluster continuously. Use continuous processing, such as a 10-second trigger, when low-latency pipelines are needed, though this requires an always-running cluster, which is typically more expensive than provisioning a new cluster periodically. _Developers weighing streaming architectures can track Spark and Delta Lake tradeoffs like these on daily.dev._ ## Similar posts on daily.dev - [Confluent Tableflow \+ Databricks Unity Catalog](https://daily.dev/posts/confluent-tableflow-databricks-unity-catalog-3l7atiiei) · Confluent Blog · 1 upvotes · 0 comments - [From Stream to Lakehouse: Kafka Ingestion with the Flink Dynamic Iceberg Sink](https://daily.dev/posts/from-stream-to-lakehouse-kafka-ingestion-with-the-flink-dynamic-iceberg-sink-pfq6etf8x) · Flink · 2 upvotes · 0 comments - [From Batch to Micro-Batch Streaming: Lessons Learned the Hard Way in a Delta Index Pipeline](https://daily.dev/posts/from-batch-to-micro-batch-streaming-lessons-learned-the-hard-way-in-a-delta-index-pipeline-wdc2knjc0) · InfoQ · 0 upvotes · 0 comments - [Approaches to Streaming Data into Apache Iceberg Tables](https://daily.dev/posts/approaches-to-streaming-data-into-apache-iceberg-tables-7z4y6jfd7) · Dremio Blog · 0 upvotes · 0 comments - [Delta Lake Explained: Boost Data Reliability in Cloud Storage](https://daily.dev/posts/delta-lake-explained-boost-data-reliability-in-cloud-storage-oechbmvm4) · databricks · 0 upvotes · 0 comments --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#kafka](https://daily.dev/tags/kafka), [#apache-spark](https://daily.dev/tags/apache-spark) [View this post on daily.dev](https://daily.dev/posts/writing-a-kafka-stream-to-delta-lake-with-spark-structured-streaming-my2gpwp39) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Writing a Kafka Stream to Delta Lake with Spark Structured Streaming","url":"https://daily.dev/posts/writing-a-kafka-stream-to-delta-lake-with-spark-structured-streaming-my2gpwp39","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/writing-a-kafka-stream-to-delta-lake-with-spark-structured-streaming-my2gpwp39"},"datePublished":"2026-08-16T08:53:25.322Z","dateModified":"2026-08-16T08:57:05.640Z","description":"A tutorial demonstrates how to ingest a Kafka stream into a Delta Lake table using Spark Structured Streaming, covering both periodic (availableNow trigger)...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0d9c235c982c0070b3dc2a32102eb540?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0d9c235c982c0070b3dc2a32102eb540?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Delta Lake Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Delta Lake Blog","logo":"https://media.daily.dev/image/upload/s--7f2q9Asf--/f_auto,q_auto/v1786870393/logos/delta-lake?_a=BAMAMicg0","url":"https://daily.dev/sources/delta-lake"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/writing-a-kafka-stream-to-delta-lake-with-spark-structured-streaming-my2gpwp39","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-engineering,kafka,apache-spark","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Delta Lake Blog","item":"https://daily.dev/sources/delta-lake"},{"@type":"ListItem","position":3,"name":"Writing a Kafka Stream to Delta Lake with Spark Structured Streaming"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/writing-a-kafka-stream-to-delta-lake-with-spark-structured-streaming-my2gpwp39#faq","mainEntity":[{"@type":"Question","name":"How do I avoid the small file problem when streaming Kafka data into a Delta Lake table?","acceptedAnswer":{"@type":"Answer","text":"Design the ingestion pipeline to minimize small file creation, then periodically run Delta Lake's OPTIMIZE command to compact them. Streaming with low latency or partitioning by a high-cardinality column like medical_code tends to produce many small files, while partitioning by something like ingestion_date avoids the issue. Predicates let you compact only a subset, such as the previous day's ingested files, keeping the operation lightweight. See how daily.dev surfaces practical Delta Lake and Spark streaming techniques for engineers managing pipeline performance."}},{"@type":"Question","name":"Should I use availableNow trigger or continuous processing for streaming Kafka data into Delta Lake with Spark?","acceptedAnswer":{"@type":"Answer","text":"Use availableNow for periodic, incremental processing when latency requirements are relaxed, since it avoids the cost of running a cluster continuously. Use continuous processing, such as a 10-second trigger, when low-latency pipelines are needed, though this requires an always-running cluster, which is typically more expensive than provisioning a new cluster periodically. Developers weighing streaming architectures can track Spark and Delta Lake tradeoffs like these on daily.dev."}}]} ```