Mercari Engineering
Read post

Safe Chunked Execution for Large-Scale Data Updates and Deletions

Large-scale database UPDATE and DELETE operations executed as single transactions risk replication lag, high load, and UNDO log bloat. Mercari's DBRE team built a general-purpose Go tool called data-updater that accepts SQL-like configuration and automatically splits operations into safe, cursor-based batches. Key features include: three operation modes (UPDATE, DELETE, NULL/archive), replica routing for SELECT queries, pessimistic locking, JOIN support, before-SQL hooks for archiving, runtime control via Unix socket (pause, resume, batch size, interval), health-check-based hibernation, auto-interval adjustment based on hibernation ratio, automatic resume via status files, and GCS-based PK source files. The tool draws inspiration from gh-ost's philosophy of runtime operational control. Source code is not open-sourced, but the full README is published so teams can build their own implementation, optionally with generative AI assistance.

    #golang#mysql
Today•17m read time•From engineering.mercari.com
Post cover image
Table of contents
Challenges with Large-Scale Data Update and Delete OperationsSolution: A General-Purpose ToolREADME.md for the General-Purpose Data Update Tool
1.7K Impressions
Mercari Engineering's image
Mercari Engineering

Mercari's platform is a resource for developers and technology enthusiasts, offering insights into e...

29 Followers

•

368 Upvotes

Would you recommend this post?

Copy link
WhatsApp
Facebook
X
New Squad
  • © 2026 Daily Dev Ltd.
  • Guidelines
  • Explore
  • Tags
  • Sources
  • Squads
  • Leaderboard