A walkthrough of building an end-to-end weather data pipeline using open-source tools. The project uses Apache Airflow for orchestration, PostgreSQL for storage, and Metabase for BI dashboards, all containerized with Docker. Data is pulled hourly from the WeatherAPI for Brazilian state capitals, processed through a multi-layer Airflow DAG, and visualized with current, historical, and forecast views in Metabase.
Table of contents
IntroductionThe environmentWhat data are we using?Interfacing With the DatabaseAirflow DAG implementationDelivering the data60.3K Impressions