--- title: Best Apache Spark posts of 2024 | daily.dev description: The most upvoted Apache Spark posts of 2024, curated by the daily.dev community. canonical: https://daily.dev/tags/apache-spark/best-of/2024 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/tags/apache-spark/best-of/2024 og:type: website og:site_name: daily.dev og:title: Best Apache Spark posts of 2024 | daily.dev og:description: The most upvoted Apache Spark posts of 2024, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of Apache Spark — 2024 1. 1 [](/posts/i-spent-8-hours-learning-the-details-of-the-apache-spark-scheduling-process--nvqkqo0dx "I spent 8 hours learning the details of the Apache Spark scheduling process.") Article Data Engineer Things · 2y I spent 8 hours learning the details of the Apache Spark scheduling process. The post delves into the details of the Apache Spark scheduling process. It covers the anatomy of a Spark job, stages, tasks, and the Directed Acyclic Graph (DAG) scheduler. It explains how SparkContext initiates scheduling, the roles of TaskScheduler and SchedulerBackend, and the concept of data locality in task execution. The post also discusses speculative execution to handle slow tasks and the entire end-to-end scheduling process in Spark. 87 2. 2 [](/posts/poc-data-platform-project-utilizing-modern-data-stack-airflow-spark-dbt-trino-hive-metastore-l-cgmnyhhkv "PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore, Lightdash, Delta Lake)") Article Data Engineer Things · 2y PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore, Lightdash, Delta Lake) The PoC Data Platform demonstrates extracting, loading, and transforming data using modern data technologies like Airflow, Spark, DBT, Trino, Hive Metastore, Lightdash, and Delta Lake. It utilizes AdventureWorks data within a data lake environment and offers insights into configuring these tools for data engineering and system design. The platform provides a comprehensive Docker setup with detailed instructions, making it a valuable resource for both beginners and professionals in data systems. 40 1 3. 3 [](/posts/storing-200-billion-entities-notion-s-data-lake-project-wvgimzklr "Storing 200 Billion Entities: Notion’s Data Lake Project") Article ByteByteGo · 2y Storing 200 Billion Entities: Notion’s Data Lake Project Notion experienced a 10-fold increase in data growth since 2021, reaching over 200 billion blocks stored in their Postgres database by 2024\. This exponential growth led to the development of a new data lake infrastructure to manage the heavy load and improve scalability, performance, and cost-efficiency. The new setup includes using S3 for storage, Kafka for data ingestion, and Apache Hudi for managing updates. This overhaul has resulted in significant cost savings, reduced ingestion times, and enabled new features such as Notion AI. 33 4. 4 [](/posts/apache-spark-in-100-seconds-dbyefe9td "Apache Spark in 100 Seconds") Video Fireship · 2y Apache Spark in 100 Seconds Apache Spark is an open-source data analytics engine designed to process massive streams of data from multiple sources at high speed by performing most tasks in memory. Created in 2009 at UC Berkeley, it is widely used in various fields, including e-commerce and space research. It supports multiple languages through APIs and can be run locally or scaled across distributed systems. Spark also has robust machine learning capabilities with its MLlib library. 29 5. 5 [](/posts/spark-beyond-basics-hidden-actions-in-your-spark-code-uvuablvz4 "Spark — Beyond Basics: Hidden actions in your spark code") Article Towards Dev · 2y Spark — Beyond Basics: Hidden actions in your spark code The post discusses hidden actions that can be mistaken for transformations in Apache Spark. It uses examples from Spark code snippets, such as \`read.csv()\`, \`df.groupby().pivot()\`, and \`foreach()\`, to explain how certain operations trigger jobs. Key insights include the impact of the inferSchema option turning a transformation into an action, and the unique behavior of pivot and foreach actions. 15 6. 6 [](/posts/6-week-free-data-engineering-boot-camp-launch-video-dataexpert-io-8zgdc8r8g "6-week Free Data Engineering Boot Camp Launch Video | DataExpert.io") Video YouTube · 2y 6-week Free Data Engineering Boot Camp Launch Video | DataExpert.io A six-week free Data Engineering Boot Camp is launching, featuring over 45 in-depth videos aimed at elevating skills in data engineering. The first two weeks cover dimensional and fact data modeling. Afterward, the boot camp splits into infrastructure and analytics tracks, addressing topics such as pipeline specs, data quality patterns, PySpark unit testing, Kafka, real-time data processing, and more. Participants can earn certificates and get hands-on assignments with AI-generated feedback, with content published daily from November 15th until the end of the year. 13 7. 7 [](/posts/i-spent-4-hours-learning-apache-spark-resource-allocation-hhioqkh7c "I spent 4 hours learning Apache Spark Resource Allocation") Article Data Engineer Things · 2y I spent 4 hours learning Apache Spark Resource Allocation An overview of Apache Spark's resource allocation mechanisms and scheduling modes. It covers static and dynamic resource allocation, highlighting how dynamic allocation uses heuristics for acquiring and removing executors. It also compares FIFO and fair scheduling, explaining how the latter ensures equal resource sharing among jobs. Additionally, considerations for gracefully decommissioning executors and the usage of an external shuffle service are discussed. 13 8. 8 [](/posts/from-data-warehousing-to-data-intelligence-how-data-took-over-zm9tgydzi "From Data Warehousing to Data Intelligence: How Data Took Over") Article databricks · 2y From Data Warehousing to Data Intelligence: How Data Took Over Organizations are moving into an era of data intelligence, using AI to understand and leverage enterprise data. This evolution was fueled by advancements like data lakehouse architecture, Apache Spark, Delta Lake, and MLflow. Over the past decade, these technologies helped break down data silos, streamline data management, and enable advanced analytics and AI. GenAI now drives this transformation further, allowing businesses to customize AI systems for their unique needs, improving both efficiency and governance in data handling. 11 9. 9 [](/posts/supernovas-black-holes-and-streaming-data-inzdaepeq "Supernovas, Black Holes and Streaming Data") Article databricks · 2y Supernovas, Black Holes and Streaming Data Learn how publicly available NASA satellite data on supernovas and black holes can be consumed and processed with Apache Kafka and Databricks. This guide covers the setup and use of Databricks Data Intelligence Platform, Delta Live Tables, and AI/BI Genie to simplify the ingestion, transformation, and visualization of streaming data. Detailed steps for using open-source Apache Spark and Kafka, along with Databricks enhancements for serverless compute and natural language querying, make this complex data accessible to data scientists and engineers. 11 [See all Apache Spark archives](/tags/apache-spark/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/tags/apache-spark/best-of/2024#page","url":"https://daily.dev/tags/apache-spark/best-of/2024","name":"Best Apache Spark Posts of 2024","description":"The most upvoted Apache Spark posts of 2024, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/tags/apache-spark/best-of/2024#items","numberOfItems":9,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/i-spent-8-hours-learning-the-details-of-the-apache-spark-scheduling-process--nvqkqo0dx","name":"I spent 8 hours learning the details of the Apache Spark scheduling process."},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/poc-data-platform-project-utilizing-modern-data-stack-airflow-spark-dbt-trino-hive-metastore-l-cgmnyhhkv","name":"PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore, Lightdash, Delta Lake)"},{"@type":"ListItem","position":3,"url":"https://daily.dev/posts/storing-200-billion-entities-notion-s-data-lake-project-wvgimzklr","name":"Storing 200 Billion Entities: Notion’s Data Lake Project"},{"@type":"ListItem","position":4,"url":"https://daily.dev/posts/apache-spark-in-100-seconds-dbyefe9td","name":"Apache Spark in 100 Seconds"},{"@type":"ListItem","position":5,"url":"https://daily.dev/posts/spark-beyond-basics-hidden-actions-in-your-spark-code-uvuablvz4","name":"Spark — Beyond Basics: Hidden actions in your spark code"},{"@type":"ListItem","position":6,"url":"https://daily.dev/posts/6-week-free-data-engineering-boot-camp-launch-video-dataexpert-io-8zgdc8r8g","name":"6-week Free Data Engineering Boot Camp Launch Video | DataExpert.io"},{"@type":"ListItem","position":7,"url":"https://daily.dev/posts/i-spent-4-hours-learning-apache-spark-resource-allocation-hhioqkh7c","name":"I spent 4 hours learning Apache Spark Resource Allocation"},{"@type":"ListItem","position":8,"url":"https://daily.dev/posts/from-data-warehousing-to-data-intelligence-how-data-took-over-zm9tgydzi","name":"From Data Warehousing to Data Intelligence: How Data Took Over"},{"@type":"ListItem","position":9,"url":"https://daily.dev/posts/supernovas-black-holes-and-streaming-data-inzdaepeq","name":"Supernovas, Black Holes and Streaming Data"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Tags","item":"https://daily.dev/tags"},{"@type":"ListItem","position":3,"name":"Apache Spark","item":"https://daily.dev/tags/apache-spark"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/tags/apache-spark/best-of"},{"@type":"ListItem","position":5,"name":"2024"}]}]} ```