--- title: Best Data Engineer Things posts — June 2025 | daily.dev description: The most upvoted Data Engineer Things posts from June 2025, curated by the daily.dev community. canonical: https://daily.dev/sources/detlife/best-of/2025/06 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/sources/detlife/best-of/2025/06 og:type: website og:site_name: daily.dev og:title: Best Data Engineer Things posts — June 2025 | daily.dev og:description: The most upvoted Data Engineer Things posts from June 2025, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of Data Engineer Things — June 2025 1. 1 [](https://daily.dev/posts/building-a-real-time-flight-data-pipeline-with-kafka-spark-and-airflow-awklylml9 "Building a Real-Time Flight Data Pipeline with Kafka, Spark, and Airflow") Article Data Engineer Things·1y Building a Real-Time Flight Data Pipeline with Kafka, Spark, and Airflow A comprehensive guide to building a real-time flight data pipeline using Kafka for streaming, Spark for processing, and Airflow for orchestration. The pipeline fetches live flight data from a custom API, streams it through Kafka to MongoDB for storage, then uses Airflow to schedule daily ETL jobs that extract landed flight information into PostgreSQL and generate CSV reports. The project includes Docker containerization, complete code examples, and demonstrates end-to-end data engineering practices from real-time ingestion to batch processing and reporting. 119 4 2. 2 [](https://daily.dev/posts/i-stopped-using-sql-for-database-management-because-i-use-the-mcp-server-with-github-copilot-in-vs--5ay1s0j3a "I Stopped Using SQL for Database Management. Because I use the MCP Server with GitHub Copilot in VS Code") Article Data Engineer Things·1y I Stopped Using SQL for Database Management. Because I use the MCP Server with GitHub Copilot in VS Code A developer demonstrates how to replace traditional SQL database management with AI-powered tools by using GitHub Copilot and GibsonAI's MCP server in VS Code. The approach allows creating complete database schemas, deploying serverless databases, and generating CRUD APIs through simple prompts, eliminating manual schema design and boilerplate code. The tutorial shows building a travel agency database schema in under 60 seconds using natural language descriptions. 85 6 3. 3 [](https://daily.dev/posts/stream-kafka-topic-to-the-iceberg-tables-with-zero-etl-xx9on98ox "Stream Kafka Topic to the Iceberg Tables with Zero-ETL") Article Data Engineer Things·1y Stream Kafka Topic to the Iceberg Tables with Zero-ETL AutoMQ introduces Table Topic, an open-source feature that automatically converts Kafka topic messages to Iceberg tables without requiring separate ETL pipelines. The solution addresses the complexity of managing Kafka-to-lakehouse data flows by handling schema management, partitioning, and upsert operations automatically. This represents an evolution from Kafka's original shared-nothing architecture to a shared-data approach, where data is accessible through both Kafka APIs and as Iceberg tables for analytics workloads. 10 [See all Data Engineer Things archives](/sources/detlife/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/sources/detlife/best-of/2025/06#page","url":"https://daily.dev/sources/detlife/best-of/2025/06","name":"Best Data Engineer Things Posts — June 2025","description":"The most upvoted Data Engineer Things posts from June 2025, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/sources/detlife/best-of/2025/06#items","numberOfItems":3,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/building-a-real-time-flight-data-pipeline-with-kafka-spark-and-airflow-awklylml9","name":"Building a Real-Time Flight Data Pipeline with Kafka, Spark, and Airflow"},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/i-stopped-using-sql-for-database-management-because-i-use-the-mcp-server-with-github-copilot-in-vs--5ay1s0j3a","name":"I Stopped Using SQL for Database Management. Because I use the MCP Server with GitHub Copilot in VS Code"},{"@type":"ListItem","position":3,"url":"https://daily.dev/posts/stream-kafka-topic-to-the-iceberg-tables-with-zero-etl-xx9on98ox","name":"Stream Kafka Topic to the Iceberg Tables with Zero-ETL"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Sources","item":"https://daily.dev/sources"},{"@type":"ListItem","position":3,"name":"Data Engineer Things","item":"https://daily.dev/sources/detlife"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/sources/detlife/best-of"},{"@type":"ListItem","position":5,"name":"June 2025"}]}]} ```