--- title: 20 Data Pipeline & ETL Terms You MUST Know Before Your... description: A rundown of 20 essential data pipeline and ETL terms grouped into five categories: data ingestion (full vs incremental load, CDC, push vs pull,... canonical: https://daily.dev/posts/20-data-pipeline-etl-terms-you-must-know-before-your-interview-jmn25ktyc twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: 20 Data Pipeline & ETL Terms You MUST Know Before Your Interview | daily.dev og:description: A rundown of 20 essential data pipeline and ETL terms grouped into five categories: data ingestion (full vs incremental load, CDC, push vs pull,... og:url: https://daily.dev/posts/20-data-pipeline-etl-terms-you-must-know-before-your-interview-jmn25ktyc og:image: https://api.daily.dev/og/posts/JMn25KtYC.png og:image:alt: 20 Data Pipeline & ETL Terms You MUST Know Before Your Interview og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # 20 Data Pipeline & ETL Terms You MUST Know Before Your Interview **[techTFQ](https://daily.dev/sources/techtfq)** · 12 min read · 1 upvotes · 0 comments ## Summary A rundown of 20 essential data pipeline and ETL terms grouped into five categories: data ingestion (full vs incremental load, CDC, push vs pull, staging/landing zones), orchestration and processing (DAGs, orchestration vs scheduling, microbatching, backfill), reliability and fault tolerance (idempotency, retry with exponential backoff, checkpointing, delivery semantics, dead letter queues), data movement and change tracking (watermarking, late arriving data, upsert/merge, deduplication), and data quality/governance (validation vs reconciliation, observability vs monitoring, data contracts). Each term includes a short explanation and a practical scenario showing why it matters. Ends with a promotion for the creator's SQL learning platform, SQL Nest, and its new free daily challenge series, 31 Days of October. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the difference between at-least-once, at-most-once, and exactly-once delivery semantics in a data pipeline? At-most-once means data might get dropped but never duplicated; at-least-once means data might get duplicated but never dropped; exactly-once means neither dropping nor duplication happens. Exactly-once is the hardest and most expensive guarantee to build, so many production pipelines settle for at-least-once combined with deduplication or upserts to stay effectively idempotent. _Engineers weighing delivery guarantees for streaming systems can dig deeper into these tradeoffs on daily.dev._ ### What is the difference between a data pipeline scheduler and an orchestrator like Airflow? A scheduler simply triggers a job at a set time, like a cron job firing at 2am, with no awareness of dependencies. An orchestrator, such as Airflow or Dagster, manages the entire dependency graph between tasks, deciding what must finish before another step starts, and handling retries or alerts when a step fails. _Anyone choosing between pipeline tools can compare orchestration options in more depth on daily.dev._ ### How do you safely restart an ETL pipeline that failed after inserting 80% of a day's records without creating duplicates? A safe restart relies on idempotency: instead of blindly rerunning the same insert statement, it either overwrites that day's entire partition or upserts using a unique key so nothing gets counted twice. This is why idempotent design matters for every retry and backfill, since without it, reruns risk quietly duplicating data. _Data engineers debugging duplicate records after a failed run can find similar fixes on daily.dev._ ## Similar posts on daily.dev - [What Is a Data Pipeline? How Data Actually Moves Through a Modern System](https://daily.dev/posts/what-is-a-data-pipeline-how-data-actually-moves-through-a-modern-system-u1tx8cvyd) · Medium · 2 upvotes · 1 comments - [What is data pipeline architecture?](https://daily.dev/posts/what-is-data-pipeline-architecture--2bbxlj9tj) · databricks · 1 upvotes · 0 comments - [ETL Process Optimization in 2026: A Practitioner's Field Guide](https://daily.dev/posts/etl-process-optimization-in-2026-a-practitioner-s-field-guide-izdncfddw) · BigData Boutique blog · 8 upvotes · 0 comments - [Embedding pipelines are the new ETL](https://daily.dev/posts/embedding-pipelines-are-the-new-etl-ww0n178fr) · InfoWorld · 0 upvotes · 0 comments --- Tags: [#backend](https://daily.dev/tags/backend), [#sql](https://daily.dev/tags/sql), [#data-engineering](https://daily.dev/tags/data-engineering), [#etl](https://daily.dev/tags/etl) [View this post on daily.dev](https://daily.dev/posts/20-data-pipeline-etl-terms-you-must-know-before-your-interview-jmn25ktyc) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"20 Data Pipeline & ETL Terms You MUST Know Before Your Interview","url":"https://daily.dev/posts/20-data-pipeline-etl-terms-you-must-know-before-your-interview-jmn25ktyc","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/20-data-pipeline-etl-terms-you-must-know-before-your-interview-jmn25ktyc"},"datePublished":"2026-09-29T17:27:39.301Z","dateModified":"2026-09-29T19:16:26.222Z","description":"A rundown of 20 essential data pipeline and ETL terms grouped into five categories: data ingestion (full vs incremental load, CDC, push vs pull,...","image":"https://i.ytimg.com/vi/vCFjyiyU0SE/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/vCFjyiyU0SE/sddefault.jpg","isAccessibleForFree":true,"articleSection":"techTFQ","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"techTFQ","logo":"https://media.daily.dev/image/upload/s--dgjhXj89--/f_auto,q_auto/v1780213385/logos/techtfq?_a=BAMAMiWQ0","url":"https://daily.dev/sources/techtfq"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/20-data-pipeline-etl-terms-you-must-know-before-your-interview-jmn25ktyc","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"backend,sql,data-engineering,etl","timeRequired":"PT12M","video":{"@type":"VideoObject","name":"20 Data Pipeline & ETL Terms You MUST Know Before Your Interview","description":"A rundown of 20 essential data pipeline and ETL terms grouped into five categories: data ingestion (full vs incremental load, CDC, push vs pull,...","thumbnailUrl":"https://i.ytimg.com/vi/vCFjyiyU0SE/sddefault.jpg","uploadDate":"2026-09-29T17:27:39.301Z","duration":"PT12M","url":"https://api.daily.dev/r/JMn25KtYC","embedUrl":"https://www.youtube.com/embed/vCFjyiyU0SE"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"techTFQ","item":"https://daily.dev/sources/techtfq"},{"@type":"ListItem","position":3,"name":"20 Data Pipeline & ETL Terms You MUST Know Before Your Interview"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/20-data-pipeline-etl-terms-you-must-know-before-your-interview-jmn25ktyc#faq","mainEntity":[{"@type":"Question","name":"What is the difference between at-least-once, at-most-once, and exactly-once delivery semantics in a data pipeline?","acceptedAnswer":{"@type":"Answer","text":"At-most-once means data might get dropped but never duplicated; at-least-once means data might get duplicated but never dropped; exactly-once means neither dropping nor duplication happens. Exactly-once is the hardest and most expensive guarantee to build, so many production pipelines settle for at-least-once combined with deduplication or upserts to stay effectively idempotent. Engineers weighing delivery guarantees for streaming systems can dig deeper into these tradeoffs on daily.dev."}},{"@type":"Question","name":"What is the difference between a data pipeline scheduler and an orchestrator like Airflow?","acceptedAnswer":{"@type":"Answer","text":"A scheduler simply triggers a job at a set time, like a cron job firing at 2am, with no awareness of dependencies. An orchestrator, such as Airflow or Dagster, manages the entire dependency graph between tasks, deciding what must finish before another step starts, and handling retries or alerts when a step fails. Anyone choosing between pipeline tools can compare orchestration options in more depth on daily.dev."}},{"@type":"Question","name":"How do you safely restart an ETL pipeline that failed after inserting 80% of a day's records without creating duplicates?","acceptedAnswer":{"@type":"Answer","text":"A safe restart relies on idempotency: instead of blindly rerunning the same insert statement, it either overwrites that day's entire partition or upserts using a unique key so nothing gets counted twice. This is why idempotent design matters for every retry and backfill, since without it, reruns risk quietly duplicating data. Data engineers debugging duplicate records after a failed run can find similar fixes on daily.dev."}}]} ```