--- title: Agentic Workloads on Airflow: Observable, Retryable, and... description: A deep-dive into building agentic LLM pipelines on Apache Airflow 3 using Dynamic Task Mapping and the apache-airflow-providers-common-ai provider. The post... canonical: https://daily.dev/posts/agentic-workloads-on-airflow-observable-retryable-and-auditable-by-design-fxdebusbq twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design | daily.dev og:description: A deep-dive into building agentic LLM pipelines on Apache Airflow 3 using Dynamic Task Mapping and the apache-airflow-providers-common-ai provider. The post... og:url: https://daily.dev/posts/agentic-workloads-on-airflow-observable-retryable-and-auditable-by-design-fxdebusbq og:image: https://api.daily.dev/og/posts/fXdEBUSBq.png og:image:alt: Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design **[Apache Airflow](https://daily.dev/sources/airflow)** · 14 min read · 1 upvotes · 0 comments ## Summary A deep-dive into building agentic LLM pipelines on Apache Airflow 3 using Dynamic Task Mapping and the apache-airflow-providers-common-ai provider. The post walks through a multi-dimensional survey analysis DAG that fans out four parallel SQL-generation LLM calls, executes them via Apache DataFusion, collects results, and synthesizes a narrative with a second LLM call — all with each step named, logged, independently retryable, and auditable via XCom. The pattern is contrasted with opaque agent harness reasoning loops, highlighting how Airflow's task boundaries enforce SQL Architect/Critic/Narrator role separation and make every intermediate artifact inspectable. The post also covers human-in-the-loop approval via ApprovalOperator and discusses the deliberate tradeoff of DAG acyclicity versus generative feedback loops. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I make LLM tool calls in an agentic pipeline individually retryable and auditable instead of hidden inside a reasoning loop? Structure each LLM call as a separate named Airflow task using Dynamic Task Mapping rather than embedding it in an agent's internal reasoning loop. With Airflow's common.ai provider, LLMSQLQueryOperator.partial().expand() creates one mapped task instance per sub-question, each with its own log, retry counter, and XCom entry, so a single failed instance (e.g. a DataFusion null-value error) can be retried without rerunning the others. _For teams wiring LLM calls into production pipelines, daily.dev surfaces patterns for making agentic workflows auditable and retryable._ ### What is the SQL Architect Critic Narrator pattern in multi-agent LLM systems? It is a separation-of-concerns pattern where distinct roles handle SQL generation (Architect), validation of output quality (Critic), and plain-language communication (Narrator), preventing one model from mixing generation, evaluation, and reporting into mediocre combined output. In an Airflow DAG built with the common.ai provider, this separation emerges naturally from task boundaries: LLMSQLQueryOperator acts as Architect, sqlglot validation plus LLMBranchOperator acts as Critic, and LLMOperator acts as Narrator. _Developers designing multi-agent LLM systems can track patterns like architect-critic-narrator separation on daily.dev._ ### Why can't an Airflow DAG implement a true feedback loop between a Critic and an Architect agent like some multi-agent frameworks do? Airflow DAGs are acyclic by design, so a Critic task cannot loop back indefinitely to an Architect task the way a multi-agent reasoning system can. Instead, the Critic either raises an exception to trigger a blunt automatic task-level retry of the Architect instance, or routes to an alternative path via LLMBranchOperator, which is explicit and auditable but requires the alternative path to be wired in ahead of time. _Engineers weighing DAG-based orchestration against agent frameworks can follow these tradeoffs on daily.dev._ ## Similar posts on daily.dev - [Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow](https://daily.dev/posts/introducing-the-common-ai-provider-llm-and-ai-agent-support-for-apache-airflow-0vn8ziyl5) · Apache Airflow · 0 upvotes · 0 comments - [Auditable Agentic Orchestration: From Autonomous Systems to Governed Execution](https://daily.dev/posts/auditable-agentic-orchestration-from-autonomous-systems-to-governed-execution-djs0eemr1) · Scott Logic · 0 upvotes · 0 comments - [Medium](https://daily.dev/posts/medium-nncxx0bwq) · Medium · 0 upvotes · 0 comments - [Durable Workflows in the Microsoft Agent Framework](https://daily.dev/posts/durable-workflows-in-the-microsoft-agent-framework-9fo8zx5rh) · .NET Blog · 0 upvotes · 0 comments - [Managed Apache Airflow: Scaling Data & AI Workloads](https://daily.dev/posts/managed-apache-airflow-scaling-data-ai-workloads-1s4kat53h) · Google Cloud · 0 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#apache-airflow](https://daily.dev/tags/apache-airflow) [View this post on daily.dev](https://daily.dev/posts/agentic-workloads-on-airflow-observable-retryable-and-auditable-by-design-fxdebusbq) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design","url":"https://daily.dev/posts/agentic-workloads-on-airflow-observable-retryable-and-auditable-by-design-fxdebusbq","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/agentic-workloads-on-airflow-observable-retryable-and-auditable-by-design-fxdebusbq"},"datePublished":"2026-07-09T07:50:07.629Z","dateModified":"2026-09-13T20:03:27.346Z","description":"A deep-dive into building agentic LLM pipelines on Apache Airflow 3 using Dynamic Task Mapping and the apache-airflow-providers-common-ai provider. The post...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8ab34af58ebe0f0f3864ba9816a1c7cb?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8ab34af58ebe0f0f3864ba9816a1c7cb?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Apache Airflow","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Apache Airflow","logo":"https://media.daily.dev/image/upload/s--Dqa6irgE--/f_auto/v1752573492/logos/airflow","url":"https://daily.dev/sources/airflow"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/agentic-workloads-on-airflow-observable-retryable-and-auditable-by-design-fxdebusbq","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,apache-airflow","timeRequired":"PT14M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Apache Airflow","item":"https://daily.dev/sources/airflow"},{"@type":"ListItem","position":3,"name":"Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/agentic-workloads-on-airflow-observable-retryable-and-auditable-by-design-fxdebusbq#faq","mainEntity":[{"@type":"Question","name":"How can I make LLM tool calls in an agentic pipeline individually retryable and auditable instead of hidden inside a reasoning loop?","acceptedAnswer":{"@type":"Answer","text":"Structure each LLM call as a separate named Airflow task using Dynamic Task Mapping rather than embedding it in an agent's internal reasoning loop. With Airflow's common.ai provider, LLMSQLQueryOperator.partial().expand() creates one mapped task instance per sub-question, each with its own log, retry counter, and XCom entry, so a single failed instance (e.g. a DataFusion null-value error) can be retried without rerunning the others. For teams wiring LLM calls into production pipelines, daily.dev surfaces patterns for making agentic workflows auditable and retryable."}},{"@type":"Question","name":"What is the SQL Architect Critic Narrator pattern in multi-agent LLM systems?","acceptedAnswer":{"@type":"Answer","text":"It is a separation-of-concerns pattern where distinct roles handle SQL generation (Architect), validation of output quality (Critic), and plain-language communication (Narrator), preventing one model from mixing generation, evaluation, and reporting into mediocre combined output. In an Airflow DAG built with the common.ai provider, this separation emerges naturally from task boundaries: LLMSQLQueryOperator acts as Architect, sqlglot validation plus LLMBranchOperator acts as Critic, and LLMOperator acts as Narrator. Developers designing multi-agent LLM systems can track patterns like architect-critic-narrator separation on daily.dev."}},{"@type":"Question","name":"Why can't an Airflow DAG implement a true feedback loop between a Critic and an Architect agent like some multi-agent frameworks do?","acceptedAnswer":{"@type":"Answer","text":"Airflow DAGs are acyclic by design, so a Critic task cannot loop back indefinitely to an Architect task the way a multi-agent reasoning system can. Instead, the Critic either raises an exception to trigger a blunt automatic task-level retry of the Architect instance, or routes to an alternative path via LLMBranchOperator, which is explicit and auditable but requires the alternative path to be wired in ahead of time. Engineers weighing DAG-based orchestration against agent frameworks can follow these tradeoffs on daily.dev."}}]} ```