A walkthrough builds a minimal Airflow pipeline that uploads local CSV/NDJSON files to S3 and loads them into Databend Cloud using COPY INTO. It covers DAG design, Airflow connections and variables, credential handling, and production hardening tips such as unique S3 keys, secrets backends, retries, and separating cross-system orchestration from in-warehouse processing.
Table of contents
Why a Reliable Ingestion Pipeline MattersThe Responsibilities of Each LayerArchitecturePrerequisitesPrepare the Sample DataStart Airflow LocallyBuild the DAGConfigure Airflow Connections and VariablesRun and Verify the PipelineProduction HardeningClosing ThoughtsFurther ReadingQuestions this post answers
How do I use Airflow to load CSV or NDJSON files from S3 into Databend Cloud?
Build a two-task Airflow DAG where the first task uploads a local file to S3 using LocalFilesystemToS3Operator, and the second task runs a COPY INTO statement against Databend Cloud via the databend-driver Python package. Chain them with upload_to_s3 >> copy_task so the load only runs after the upload succeeds, and store credentials in Airflow Variables or Connections rather than hardcoding them. daily.dev surfaces practical data engineering patterns for teams wiring Airflow into cloud warehouses.
How can I make S3-to-warehouse ingestion pipelines safe to retry without creating duplicate rows?
Use unique S3 object keys per batch or partition, such as ingest/dt=2026-06-29/hour=10/sample.ndjson, instead of overwriting the same key. This keeps lineage clear and removes ambiguity around retries and backfills. Also set PURGE=FALSE so the source file stays in S3 after loading, and ON_ERROR=ABORT so bad input fails loudly rather than skipping rows silently. developers hardening ingestion pipelines track patterns like this for retries and deduplication on daily.dev.