--- title: Best Data Engineer Things posts — October 2024 | daily.dev description: The most upvoted Data Engineer Things posts from October 2024, curated by the daily.dev community. canonical: https://daily.dev/sources/detlife/best-of/2024/10 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/sources/detlife/best-of/2024/10 og:type: website og:site_name: daily.dev og:title: Best Data Engineer Things posts — October 2024 | daily.dev og:description: The most upvoted Data Engineer Things posts from October 2024, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of Data Engineer Things — October 2024 1. 1 [](https://daily.dev/posts/i-spent-8-hours-learning-the-details-of-the-apache-spark-scheduling-process--nvqkqo0dx "I spent 8 hours learning the details of the Apache Spark scheduling process.") Article Data Engineer Things · 2y I spent 8 hours learning the details of the Apache Spark scheduling process. The post delves into the details of the Apache Spark scheduling process. It covers the anatomy of a Spark job, stages, tasks, and the Directed Acyclic Graph (DAG) scheduler. It explains how SparkContext initiates scheduling, the roles of TaskScheduler and SchedulerBackend, and the concept of data locality in task execution. The post also discusses speculative execution to handle slow tasks and the entire end-to-end scheduling process in Spark. 87 2. 2 [](https://daily.dev/posts/i-spent-6-hours-learning-apache-arrow-overview-gg8sy4pzy "I spent 6 hours learning Apache Arrow: Overview") Article Data Engineer Things · 2y I spent 6 hours learning Apache Arrow: Overview Apache Arrow is a standard memory format designed for efficient data processing in analytics workloads. It focuses on performance and interoperability by leveraging a columnar in-memory format and aligned memory allocation. Arrow minimizes serialization and deserialization costs, enabling efficient data sharing between systems. Key elements include physical memory layouts for arrays, record batch serialization, and IPC formats enabling seamless inter-process and network data transfers. Arrow is widely adopted by various data projects, enhancing their performance and data handling capabilities. 51 2 3. 3 [](https://daily.dev/posts/poc-data-platform-project-utilizing-modern-data-stack-airflow-spark-dbt-trino-hive-metastore-l-cgmnyhhkv "PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore, Lightdash, Delta Lake)") Article Data Engineer Things · 2y PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore, Lightdash, Delta Lake) The PoC Data Platform demonstrates extracting, loading, and transforming data using modern data technologies like Airflow, Spark, DBT, Trino, Hive Metastore, Lightdash, and Delta Lake. It utilizes AdventureWorks data within a data lake environment and offers insights into configuring these tools for data engineering and system design. The platform provides a comprehensive Docker setup with detailed instructions, making it a valuable resource for both beginners and professionals in data systems. 40 1 [See all Data Engineer Things archives](/sources/detlife/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/sources/detlife/best-of/2024/10#page","url":"https://daily.dev/sources/detlife/best-of/2024/10","name":"Best Data Engineer Things Posts — October 2024","description":"The most upvoted Data Engineer Things posts from October 2024, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/sources/detlife/best-of/2024/10#items","numberOfItems":3,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/i-spent-8-hours-learning-the-details-of-the-apache-spark-scheduling-process--nvqkqo0dx","name":"I spent 8 hours learning the details of the Apache Spark scheduling process."},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/i-spent-6-hours-learning-apache-arrow-overview-gg8sy4pzy","name":"I spent 6 hours learning Apache Arrow: Overview"},{"@type":"ListItem","position":3,"url":"https://daily.dev/posts/poc-data-platform-project-utilizing-modern-data-stack-airflow-spark-dbt-trino-hive-metastore-l-cgmnyhhkv","name":"PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore, Lightdash, Delta Lake)"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Sources","item":"https://daily.dev/sources"},{"@type":"ListItem","position":3,"name":"Data Engineer Things","item":"https://daily.dev/sources/detlife"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/sources/detlife/best-of"},{"@type":"ListItem","position":5,"name":"October 2024"}]}]} ```