Most data pipeline backfills are preventable, not inevitable. The post categorizes backfills into three types: bug backfills (Type A, avoidable), schema-change backfills (Type B), and new data reconstruction (Type C). Type A backfills — the most common — stem from five root causes: non-idempotent pipelines, no raw data replay layer, missing schema validation at ingest, downstream tables that have drifted from their inputs via manual edits, and absent correctness monitoring. A practical test is proposed: measure how long and how many people it takes to regenerate a partition from three months ago. The answer reveals whether a team has a real data platform or just a collection of scripts that happened to work once.
Table of contents
Everybody calls backfills a "necessary evil"Three kinds of backfillsFive reasons you needed that backfillThe one test worth runningSix properties6.3K Impressions