Halodoc's data engineering team shares how they migrated from Airflow 2.10 to 3.2.1 on Amazon MWAA to reduce resource waste during ETL burst windows. Two key changes drove the gains: replacing polling sensors with Airflow 3.0 Data Assets (event-driven scheduling that eliminates worker slots held just to wait), and swapping a synchronous psycopg-based Redshift operator for the deferrable RedshiftDataOperator backed by the Redshift Data API. The migration covered 30 DAGs for Data Assets and 130 for the deferrable operator. Results after a month of monitoring: base worker CPU dropped from 26.1% to 7.71%, memory from 49.2% to 30.8%, scheduler CPU from 36.1% to 29.4%, and table-locking errors fell ~38%. The post also details a critical N+1 query bug in Airflow 3.0.6 that pinned the metadata DB at 100% CPU, requiring an upgrade to 3.2.1, plus a DAG versioning pitfall caused by runtime-variable values in DAG constructors.
