--- title: Best Apache Spark posts of 2024 | daily.dev description: The most upvoted Apache Spark posts of 2024, curated by the daily.dev community. canonical: https://daily.dev/tags/spark/best-of/2024 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/tags/spark/best-of/2024 og:type: website og:site_name: daily.dev og:title: Best Apache Spark posts of 2024 | daily.dev og:description: The most upvoted Apache Spark posts of 2024, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of Apache Spark — 2024 1. 1 [](/posts/cloud-native-data-engineering-orchestrating-spark-on-kubernetes-with-custom-airflow-operator-and-gc-mxn05ogcg "Cloud-Native Data Engineering: Orchestrating Spark on Kubernetes with Custom Airflow Operator and GCS Integration") Article Data Engineer Things · 2y Cloud-Native Data Engineering: Orchestrating Spark on Kubernetes with Custom Airflow Operator and GCS Integration This guide provides step-by-step instructions for setting up a scalable, automated data pipeline using Spark on Kubernetes with Google Cloud Storage (GCS) integration, managed by Apache Airflow. It includes configuring custom Docker images for Spark with GCS support, installing and configuring the Spark Operator and Airflow, creating a custom Airflow operator for submitting Spark jobs, and setting up necessary role-based access controls. By the end, readers will have a robust, cloud-native data engineering platform capable of handling complex data workflows efficiently. 16 2. 2 [](/posts/spark-beyond-basics-hidden-actions-in-your-spark-code-uvuablvz4 "Spark — Beyond Basics: Hidden actions in your spark code") Article Towards Dev · 2y Spark — Beyond Basics: Hidden actions in your spark code The post discusses hidden actions that can be mistaken for transformations in Apache Spark. It uses examples from Spark code snippets, such as \`read.csv()\`, \`df.groupby().pivot()\`, and \`foreach()\`, to explain how certain operations trigger jobs. Key insights include the impact of the inferSchema option turning a transformation into an action, and the unique behavior of pivot and foreach actions. 15 3. 3 [](/posts/a-day-in-the-life-of-a-data-engineer-at-agoda-skuyhqhgv "A Day in the Life of a Data Engineer at Agoda") Article agoda · 2y A Day in the Life of a Data Engineer at Agoda At Agoda, data fuels every decision and data engineers play a key role by designing and maintaining data pipelines. Lookuut Struchkov, a Staff Data Engineer, discusses his journey and daily responsibilities, including optimizing data pipelines, collaborating with various teams, and handling on-call support requests. He emphasizes the importance of continuous learning and shares advice for aspiring data engineers. Agoda stands out for its commitment to innovation, use of cutting-edge technologies like Spark and Scala, and its collaborative culture. 14 1 [See all Apache Spark archives](/tags/spark/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/tags/spark/best-of/2024#page","url":"https://daily.dev/tags/spark/best-of/2024","name":"Best Apache Spark Posts of 2024","description":"The most upvoted Apache Spark posts of 2024, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/tags/spark/best-of/2024#items","numberOfItems":3,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/cloud-native-data-engineering-orchestrating-spark-on-kubernetes-with-custom-airflow-operator-and-gc-mxn05ogcg","name":"Cloud-Native Data Engineering: Orchestrating Spark on Kubernetes with Custom Airflow Operator and GCS Integration"},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/spark-beyond-basics-hidden-actions-in-your-spark-code-uvuablvz4","name":"Spark — Beyond Basics: Hidden actions in your spark code"},{"@type":"ListItem","position":3,"url":"https://daily.dev/posts/a-day-in-the-life-of-a-data-engineer-at-agoda-skuyhqhgv","name":"A Day in the Life of a Data Engineer at Agoda"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Tags","item":"https://daily.dev/tags"},{"@type":"ListItem","position":3,"name":"Apache Spark","item":"https://daily.dev/tags/spark"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/tags/spark/best-of"},{"@type":"ListItem","position":5,"name":"2024"}]}]} ```