Large-scale database UPDATE and DELETE operations executed as single transactions risk replication lag, high load, and UNDO log bloat. Mercari's DBRE team built a general-purpose Go tool called data-updater that accepts SQL-like configuration and automatically splits operations into safe, cursor-based batches. Key features include: three operation modes (UPDATE, DELETE, NULL/archive), replica routing for SELECT queries, pessimistic locking, JOIN support, before-SQL hooks for archiving, runtime control via Unix socket (pause, resume, batch size, interval), health-check-based hibernation, auto-interval adjustment based on hibernation ratio, automatic resume via status files, and GCS-based PK source files. The tool draws inspiration from gh-ost's philosophy of runtime operational control. Source code is not open-sourced, but the full README is published so teams can build their own implementation, optionally with generative AI assistance.

17m read timeFrom engineering.mercari.com
Post cover image
Table of contents
Challenges with Large-Scale Data Update and Delete OperationsSolution: A General-Purpose ToolREADME.md for the General-Purpose Data Update Tool
3.5K Impressions