--- title: The ETL Pipeline Handbook: How to Build a... description: A comprehensive hands-on tutorial for building a production-grade ETL pipeline in Python using real-world flood monitoring data from France's Hub'Eau API.... canonical: https://daily.dev/posts/the-etl-pipeline-handbook-how-to-build-a-production-grade-pipeline-in-python-zsxxadgsx twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The ETL Pipeline Handbook: How to Build a Production-Grade Pipeline in Python | daily.dev og:description: A comprehensive hands-on tutorial for building a production-grade ETL pipeline in Python using real-world flood monitoring data from France's Hub'Eau API.... og:url: https://daily.dev/posts/the-etl-pipeline-handbook-how-to-build-a-production-grade-pipeline-in-python-zsxxadgsx og:image: https://api.daily.dev/og/posts/zSXxadGsX.png og:image:alt: The ETL Pipeline Handbook: How to Build a Production-Grade Pipeline in Python og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The ETL Pipeline Handbook: How to Build a Production-Grade Pipeline in Python **[freeCodeCamp](https://daily.dev/sources/freecodecamp)** · 44 min read · 3 upvotes · 0 comments ## Summary A comprehensive hands-on tutorial for building a production-grade ETL pipeline in Python using real-world flood monitoring data from France's Hub'Eau API. Covers the full pipeline lifecycle: extracting data from a paginated REST API (with mock support for safe testing), transforming it with pandas using graceful type coercion, schema translation from French to English, derived column computation, deduplication for idempotency, and incremental loading to avoid re-fetching historical data. Key engineering patterns include using Python dataclasses for centralized configuration, the Null Object pattern for graceful file loading, Single Responsibility Principle for fetch functions, and immutable DataFrame transforms via df.copy(). The pipeline is designed to run unattended on a weekly schedule and publishes results to Kaggle. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I avoid the mutable default argument bug when using Python dataclasses? Use field(default_factory=...) instead of assigning a mutable literal like a list directly as a default value. A plain default such as station_codes: List[str] = [] causes every instance to share the same underlying list object, so mutating it on one instance mutates it everywhere. field(default_factory=list) or a lambda factory creates a fresh, independent object for each new instance. _Developers wiring up Python dataclasses can find gotchas like these surfaced alongside other python content on daily.dev._ ### How should I design an ETL pipeline to only fetch new data instead of re-downloading everything each run? Check the most recent date already stored in the dataset, then request only records from the day after that date onward, merging new rows into the existing dataset. The check compares against yesterday's date rather than today, since same-day API observations may not be finalized yet. If the dataset already covers yesterday, no fetch is needed at all. _Teams building incremental data pipelines can track patterns like this through daily.dev's data engineering coverage._ ### What's a reliable way to deduplicate rows in an incremental data pipeline built with pandas? Build a composite key from the fields that together define uniqueness, such as station code, observation date, and measured value, then compare a set of existing keys against the new data's keys using .isin() for fast membership checks. Using a Python set instead of a list for lookups is O(1) versus O(n), which matters as the dataset grows to tens of thousands of rows. _Developers deciding how to keep scheduled pandas pipelines idempotent can dig deeper into these patterns via daily.dev._ ## Similar posts on daily.dev - [I Built My First ETL Pipeline as a Complete Beginner. Here’s How.](https://daily.dev/posts/i-built-my-first-etl-pipeline-as-a-complete-beginner-here-s-how--u9lbb6ejz) · Towards Data Science · 0 upvotes · 0 comments - [Efficient Data Processing in Python: Batch vs Streaming Pipelines Explained](https://daily.dev/posts/efficient-data-processing-in-python-batch-vs-streaming-pipelines-explained-3bo6fa2lr) · freeCodeCamp · 19 upvotes · 0 comments - [I Built My Second ETL Pipeline. This Time, I Started Thinking Like a Data Engineer](https://daily.dev/posts/i-built-my-second-etl-pipeline-this-time-i-started-thinking-like-a-data-engineer-3mf1pyvke) · Towards Data Science · 7 upvotes · 0 comments --- Tags: [#python](https://daily.dev/tags/python), [#backend](https://daily.dev/tags/backend), [#data-engineering](https://daily.dev/tags/data-engineering), [#pandas](https://daily.dev/tags/pandas), [#etl](https://daily.dev/tags/etl) [View this post on daily.dev](https://daily.dev/posts/the-etl-pipeline-handbook-how-to-build-a-production-grade-pipeline-in-python-zsxxadgsx) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"The ETL Pipeline Handbook: How to Build a Production-Grade Pipeline in Python","url":"https://daily.dev/posts/the-etl-pipeline-handbook-how-to-build-a-production-grade-pipeline-in-python-zsxxadgsx","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/the-etl-pipeline-handbook-how-to-build-a-production-grade-pipeline-in-python-zsxxadgsx"},"datePublished":"2026-07-27T21:32:40.259Z","dateModified":"2026-09-13T20:06:39.204Z","description":"A comprehensive hands-on tutorial for building a production-grade ETL pipeline in Python using real-world flood monitoring data from France's Hub'Eau API....","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/01b558943f3d44d55f089512eb7b664c?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/01b558943f3d44d55f089512eb7b664c?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"freeCodeCamp","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"freeCodeCamp","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1628412854/logos/freecodecamp","url":"https://daily.dev/sources/freecodecamp"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/the-etl-pipeline-handbook-how-to-build-a-production-grade-pipeline-in-python-zsxxadgsx","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,backend,data-engineering,pandas,etl","timeRequired":"PT44M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"freeCodeCamp","item":"https://daily.dev/sources/freecodecamp"},{"@type":"ListItem","position":3,"name":"The ETL Pipeline Handbook: How to Build a Production-Grade Pipeline in Python"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/the-etl-pipeline-handbook-how-to-build-a-production-grade-pipeline-in-python-zsxxadgsx#faq","mainEntity":[{"@type":"Question","name":"How do I avoid the mutable default argument bug when using Python dataclasses?","acceptedAnswer":{"@type":"Answer","text":"Use field(default_factory=...) instead of assigning a mutable literal like a list directly as a default value. A plain default such as station_codes: List[str] = [] causes every instance to share the same underlying list object, so mutating it on one instance mutates it everywhere. field(default_factory=list) or a lambda factory creates a fresh, independent object for each new instance. Developers wiring up Python dataclasses can find gotchas like these surfaced alongside other python content on daily.dev."}},{"@type":"Question","name":"How should I design an ETL pipeline to only fetch new data instead of re-downloading everything each run?","acceptedAnswer":{"@type":"Answer","text":"Check the most recent date already stored in the dataset, then request only records from the day after that date onward, merging new rows into the existing dataset. The check compares against yesterday's date rather than today, since same-day API observations may not be finalized yet. If the dataset already covers yesterday, no fetch is needed at all. Teams building incremental data pipelines can track patterns like this through daily.dev's data engineering coverage."}},{"@type":"Question","name":"What's a reliable way to deduplicate rows in an incremental data pipeline built with pandas?","acceptedAnswer":{"@type":"Answer","text":"Build a composite key from the fields that together define uniqueness, such as station code, observation date, and measured value, then compare a set of existing keys against the new data's keys using .isin() for fast membership checks. Using a Python set instead of a list for lookups is O(1) versus O(n), which matters as the dataset grows to tens of thousands of rows. Developers deciding how to keep scheduled pandas pipelines idempotent can dig deeper into these patterns via daily.dev."}}]} ```