--- title: Best Apache Airflow posts of 2024 | daily.dev description: The most upvoted Apache Airflow posts of 2024, curated by the daily.dev community. canonical: https://daily.dev/tags/apache-airflow/best-of/2024 twitter:card: summary_large_image twitter:site: @dailydotdev og:url: https://daily.dev/tags/apache-airflow/best-of/2024 og:type: website og:site_name: daily.dev og:title: Best Apache Airflow posts of 2024 | daily.dev og:description: The most upvoted Apache Airflow posts of 2024, curated by the daily.dev community. og:image: https://media.daily.dev/image/upload/s--VAY5ToZt--/f_auto/v1724209435/public/daily.dev%20-%20open%20graph --- # Best of Apache Airflow — 2024 1. 1 [](/posts/i-spent-5-hours-learning-how-clickhouse-built-their-internal-data-warehouse--ykgna2cg3 "I spent 5 hours learning how ClickHouse built their internal data warehouse.") Article Data Engineer Things · 2y I spent 5 hours learning how ClickHouse built their internal data warehouse. ClickHouse has built an internal data warehouse to handle 50 TB of data daily, incorporating multiple internal sources like AWS, GCP, and Salesforce. They use Airflow for scheduling, AWS S3 as the intermediate data layer, and Superset for BI tools. Key features include data consistency, idempotency, and real-time analytics. They have also adopted dbt to streamline data transformations and introduced new tools for improved user access to data. 42 2. 2 [](/posts/poc-data-platform-project-utilizing-modern-data-stack-airflow-spark-dbt-trino-hive-metastore-l-cgmnyhhkv "PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore, Lightdash, Delta Lake)") Article Data Engineer Things · 2y PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore, Lightdash, Delta Lake) The PoC Data Platform demonstrates extracting, loading, and transforming data using modern data technologies like Airflow, Spark, DBT, Trino, Hive Metastore, Lightdash, and Delta Lake. It utilizes AdventureWorks data within a data lake environment and offers insights into configuring these tools for data engineering and system design. The platform provides a comprehensive Docker setup with detailed instructions, making it a valuable resource for both beginners and professionals in data systems. 40 1 3. 3 [](/posts/how-to-set-up-apache-airflow-for-10-minutes-via-docker-6dqaqx3xh "How to set up Apache Airflow for 10 minutes via Docker") Article habr · 2y How to set up Apache Airflow for 10 minutes via Docker Learn how to set up Apache Airflow using Docker in just 10 minutes. The guide includes installing necessary VSCode extensions, creating the required directory and files (\`docker-compose.yml\` and \`dockerfile\`), and building a custom Docker image. Detailed steps are provided to help you see the Airflow UI running on your localhost. Instructions also cover saving strategies and final steps to Compose Down after work. 27 2 4. 4 [](/posts/data-driven-football-insights-from-web-scraping-to-visualization-using-airflow-dbt-cloud-and-aws-nbt95tkr2 "“Data-Driven Football Insights: From Web Scraping to Visualization Using Airflow, Dbt Cloud, and AWS Tech Stack”") Article Towards Dev · 2y “Data-Driven Football Insights: From Web Scraping to Visualization Using Airflow, Dbt Cloud, and AWS Tech Stack” This project automates the process of collecting, storing, and analyzing football data using technologies like Apache Airflow, DBT Cloud, and AWS. The workflow includes web scraping data using Python, storing it in Amazon S3, processing it in Amazon Redshift, transforming data with DBT Cloud, and visualizing it through Amazon QuickSight. This integrated approach offers a scalable solution to manage and analyze detailed football statistics efficiently. 24 5. 5 [](/posts/creating-an-etl-data-pipeline-using-bash-with-apache-airflow-l5fok7ru2 "Creating an ETL Data Pipeline Using Bash with Apache Airflow") Article Python in Plain English · 2y Creating an ETL Data Pipeline Using Bash with Apache Airflow Learn how to create an ETL data pipeline using bash with Apache Airflow. Extract data from various file formats, transform it, and load it into a new file. Includes steps for starting Apache Airflow, downloading the dataset, creating a DAG, and executing the pipeline. 24 6. 6 [](/posts/building-etl-pipeline-with-airflow-spotify-zuul1m8bn "Building ETL pipeline with Airflow + Spotify") Article Towards Dev · 2y Building ETL pipeline with Airflow + Spotify Learn how to build an ETL pipeline with Airflow and Spotify. Create a Spotify app and generate auth keys and tokens for API consumption. Install Airflow with Docker or Astro CLI. The pipeline includes tasks for exchanging tokens, refreshing tokens, and getting songs from Spotify API. 21 7. 7 [](/posts/learn-the-essentials-of-data-engineering-3zewvi6mu "Learn the Essentials of Data Engineering") Article freeCodeCamp · 3y Learn the Essentials of Data Engineering Learn the essentials of data engineering, including the importance of data engineering, key areas covered in the course, and the role of Docker in data engineering. 21 8. 8 [](/posts/talk-to-airflow-build-an-ai-agent-using-pydanticai-and-gemini-2-0-hgrygrcwe "Talk to Airflow — Build an AI Agent Using PydanticAI and Gemini 2.0") Article Data Engineer Things · 2y Talk to Airflow — Build an AI Agent Using PydanticAI and Gemini 2.0 Developing AI applications can be challenging, but tools like PydanticAI and models like Gemini 2.0 are making it easier. This post guides you through building an AI agent that interacts with Apache Airflow using PydanticAI, highlighting its simplicity and production-readiness. The AI agent can understand natural language queries, fetch real-time status updates, and provide structured data, similar to how modern cockpits give pilots full control over their aircraft. Examples and detailed steps showcase how to set up the environment, create sample DAGs, and implement an AI agent from scratch. 18 9. 9 [](/posts/cloud-native-data-engineering-orchestrating-spark-on-kubernetes-with-custom-airflow-operator-and-gc-mxn05ogcg "Cloud-Native Data Engineering: Orchestrating Spark on Kubernetes with Custom Airflow Operator and GCS Integration") Article Data Engineer Things · 2y Cloud-Native Data Engineering: Orchestrating Spark on Kubernetes with Custom Airflow Operator and GCS Integration This guide provides step-by-step instructions for setting up a scalable, automated data pipeline using Spark on Kubernetes with Google Cloud Storage (GCS) integration, managed by Apache Airflow. It includes configuring custom Docker images for Spark with GCS support, installing and configuring the Spark Operator and Airflow, creating a custom Airflow operator for submitting Spark jobs, and setting up necessary role-based access controls. By the end, readers will have a robust, cloud-native data engineering platform capable of handling complex data workflows efficiently. 16 10. 10 [](/posts/getting-started-with-apache-airflow-for-data-workflows-svnzch7ce "Getting Started with Apache Airflow for Data Workflows") Article Dev Genius · 2y Getting Started with Apache Airflow for Data Workflows Apache Airflow is an open-source tool used for orchestrating complex computational workflows and pipelines. It allows users to define workflows as code in Python, scale horizontally, and monitor processes through a web-based UI. Key components include Directed Acyclic Graphs (DAGs), a scheduler, executors, and a metadata database. Installation can be simplified using Docker, and various operators, sensors, and hooks enhance functionality. Features like data-aware scheduling, conditional branching, and trigger rules make Airflow a powerful tool for managing data workflows efficiently. 11 11. 11 [](/posts/airflow-3-0-promises-us-real-event-driven-scheduling-ks6vyocln "Airflow 3.0 promises us REAL event driven scheduling") Article Data Engineer Things · 2y Airflow 3.0 promises us REAL event driven scheduling Airflow 3.0 is set to introduce real event driven scheduling, featuring poll based mechanisms to enhance the current scheduling capabilities. This update aims to improve the responsiveness of Airflow to external data events, allowing for more dynamic and efficient workflow automation. The current limitations of sensors and data-aware scheduling are addressed, promising more seamless integration with systems like Snowflake and Google Cloud Storage. 10 [See all Apache Airflow archives](/tags/apache-airflow/best-of) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@graph":[{"@type":"CollectionPage","@id":"https://daily.dev/tags/apache-airflow/best-of/2024#page","url":"https://daily.dev/tags/apache-airflow/best-of/2024","name":"Best Apache Airflow Posts of 2024","description":"The most upvoted Apache Airflow posts of 2024, curated by the daily.dev community.","isPartOf":{"@type":"WebSite","url":"https://daily.dev"}},{"@type":"ItemList","@id":"https://daily.dev/tags/apache-airflow/best-of/2024#items","numberOfItems":11,"itemListElement":[{"@type":"ListItem","position":1,"url":"https://daily.dev/posts/i-spent-5-hours-learning-how-clickhouse-built-their-internal-data-warehouse--ykgna2cg3","name":"I spent 5 hours learning how ClickHouse built their internal data warehouse."},{"@type":"ListItem","position":2,"url":"https://daily.dev/posts/poc-data-platform-project-utilizing-modern-data-stack-airflow-spark-dbt-trino-hive-metastore-l-cgmnyhhkv","name":"PoC Data Platform project utilizing modern data stack (Airflow, Spark, DBT, Trino, Hive metastore, Lightdash, Delta Lake)"},{"@type":"ListItem","position":3,"url":"https://daily.dev/posts/how-to-set-up-apache-airflow-for-10-minutes-via-docker-6dqaqx3xh","name":"How to set up Apache Airflow for 10 minutes via Docker"},{"@type":"ListItem","position":4,"url":"https://daily.dev/posts/data-driven-football-insights-from-web-scraping-to-visualization-using-airflow-dbt-cloud-and-aws-nbt95tkr2","name":"“Data-Driven Football Insights: From Web Scraping to Visualization Using Airflow, Dbt Cloud, and AWS Tech Stack”"},{"@type":"ListItem","position":5,"url":"https://daily.dev/posts/creating-an-etl-data-pipeline-using-bash-with-apache-airflow-l5fok7ru2","name":"Creating an ETL Data Pipeline Using Bash with Apache Airflow"},{"@type":"ListItem","position":6,"url":"https://daily.dev/posts/building-etl-pipeline-with-airflow-spotify-zuul1m8bn","name":"Building ETL pipeline with Airflow + Spotify"},{"@type":"ListItem","position":7,"url":"https://daily.dev/posts/learn-the-essentials-of-data-engineering-3zewvi6mu","name":"Learn the Essentials of Data Engineering"},{"@type":"ListItem","position":8,"url":"https://daily.dev/posts/talk-to-airflow-build-an-ai-agent-using-pydanticai-and-gemini-2-0-hgrygrcwe","name":"Talk to Airflow — Build an AI Agent Using PydanticAI and Gemini 2.0"},{"@type":"ListItem","position":9,"url":"https://daily.dev/posts/cloud-native-data-engineering-orchestrating-spark-on-kubernetes-with-custom-airflow-operator-and-gc-mxn05ogcg","name":"Cloud-Native Data Engineering: Orchestrating Spark on Kubernetes with Custom Airflow Operator and GCS Integration"},{"@type":"ListItem","position":10,"url":"https://daily.dev/posts/getting-started-with-apache-airflow-for-data-workflows-svnzch7ce","name":"Getting Started with Apache Airflow for Data Workflows"},{"@type":"ListItem","position":11,"url":"https://daily.dev/posts/airflow-3-0-promises-us-real-event-driven-scheduling-ks6vyocln","name":"Airflow 3.0 promises us REAL event driven scheduling"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Tags","item":"https://daily.dev/tags"},{"@type":"ListItem","position":3,"name":"Apache Airflow","item":"https://daily.dev/tags/apache-airflow"},{"@type":"ListItem","position":4,"name":"Best of","item":"https://daily.dev/tags/apache-airflow/best-of"},{"@type":"ListItem","position":5,"name":"2024"}]}]} ```