Temporal Cloud migrated its billing data store from a legacy warehouse to ClickHouse, gaining three to five years of runway (up from roughly one year), cutting steady-state costs by ~70%, and delivering usage data to customers about 30 minutes sooner. The core challenge wasn't moving data but validating correctness for billing-critical queries. The team ran both systems in parallel for six weeks, shadowing every query and carefully classifying differences as tolerated (floating-point variance), expected (ClickHouse returning fresher data), or blocking (real bugs). This process caught a critical integer overflow: ClickHouse sums 64-bit integers into a same-width accumulator without auto-widening, causing negative totals for large accounts. The fix was casting before aggregating. The rollout was ordered by cost of reversal — read APIs first, billing aggregation last — and shadowing was deliberately started on high-volume accounts to surface edge cases that small accounts would never trigger.
Table of contents
How the pipeline works #Why we moved #Why code review wasn't enough #Deciding which differences were bugs #The defect this caught #Cutting over in order of reversibility #What it bought us #What we'd do again #Questions this post answers
Does ClickHouse automatically widen integer accumulators when summing large 64-bit integer columns?
No, ClickHouse sums integer columns into an integer accumulator of the same width without automatic widening. Summing 64-bit integer values across a large enough window can overflow and wrap into negative numbers. The fix is to cast the column to a wider type before aggregating, and this defect is invisible on small datasets — it only surfaces at production scale with very large accounts. Engineers migrating billing or analytics queries to ClickHouse track edge cases like this on daily.dev.
What were the measured results of migrating from a legacy data warehouse to ClickHouse Cloud for a billing workload?
Migrating Temporal Cloud's billing store to ClickHouse Cloud yielded roughly 70% lower steady-state costs (vs. a 30–50% target), three to five years of growth headroom (up from about one year), ~1.4 seconds faster average query latency across a week of live traffic, and usage data reaching customers about 30 minutes sooner due to eliminating a batch deduplication job. A schema change on a billions-row table completed in ~5 minutes with no maintenance window. Teams evaluating ClickHouse for cost or scale reasons find real migration benchmarks like these on daily.dev.
How should I order a database migration rollout to minimize risk when the pipeline has multiple read paths?
Roll out in reverse order of data-flow, prioritizing paths by cost of reversal. Read-only, per-account APIs go first because a wrong answer affects one account and is fixed by a flag change. Aggregation paths whose output leaves the system go last, after weeks of shadow validation. For shadow testing, start with the largest, highest-volume accounts — small accounts never trigger overflow or edge-case failures that only appear at scale. Developers planning staged database cutovers find architecture war stories like this on daily.dev.