Grab's Counter Service handles billions of fraud-detection queries daily. This post details a zero-downtime migration from a wide-column database to Aerospike, covering three major workstreams: decoupling the reader's storage layer in Rust using enum dispatch and a concrete facade (avoiding trait objects for performance), redesigning the writer's data model from row-per-bucket to a map-based schema that collapses all bucket counts into a single record per counter, and implementing dual read-write shadow paths for gradual traffic rollout with data parity validation. The map-based schema reduced on-disk storage from ~3 TB to ~1 TB, cut p99 read latency by ~50%, and lowered per-node costs by 45–50%. Key learnings include the Aerospike Rust client's sync-only limitation requiring spawn_blocking workarounds, a DNS re-resolution bug during cluster topology changes, and the trade-offs of NVMe-backed primary index under hot-key I/O contention.
