Agoda's engineering team describes how they eliminated configuration-related production outages by replacing direct Consul UI edits with a GitOps workflow. With over 34,000 KV pairs across four global data centers and 100+ daily changes, unvalidated edits were causing hard-to-trace incidents. The solution involved a Consul KV Sync service that watches a Git repository, validates changes against auto-generated JSON Schemas, and applies them to Consul. Migration of all 34,000 KVs was fully automated, including schema generation and CODEOWNERS assignment. The result: 100% audit trail, zero configuration-related outages post-migration, and full adoption across 100+ engineering teams. Future plans include gradual per-datacenter rollouts and distributed ownership management.
