Shopify replaced a fragmented, unpredictable multi-vendor observability stack with an internal platform called Observe, built entirely on self-hosted ClickHouse. The migration delivered a 16x query improvement out of the box (peaking past 30x) while ingesting around 100 million events per second (~110 GB/s) at Black Friday-Cyber Monday peak, keeping data queryable in under a minute. Engineering director Elijah McPherson details techniques used at this scale: buffering telemetry through Kafka before durable batch commits, using materialized views for fast autocomplete and cross-signal correlation, promoting hot Map(String,String) keys to typed columns for ~30% faster queries, sorting map keys for 20-40% disk savings, and running an in-house Kubernetes operator to manage schema migrations across ~20 tenants. Looking ahead, Shopify is moving to tiered storage, ARM-based compute, ClickHouse's native JSON type, and a Rust-based ingestion pipeline, while also exploring an AI agent layer and merchant-facing analytics. McPherson notes that if starting today, Shopify would likely choose ClickHouse Cloud instead of self-hosting to offload operational burden.
Table of contents
A disjointed, unpredictable setup #Observability built by Shopify, for Shopify #Choosing ClickHouse as the engine #Lessons from running ClickHouse at scale #Operating it day to day in production #What’s next for Shopify and ClickHouse #DIY ClickHouse vs. ClickHouse Cloud #Questions this post answers
How much of a query performance improvement did Shopify see after moving observability data to ClickHouse?
Shopify saw a 16x query performance improvement out of the box after deploying ClickHouse for observability, with performance spiking past 30x at peak. The platform now ingests roughly 50 million events per second at steady state and up to 100 million events per second (about 110 GB/s) during Black Friday-Cyber Monday peak, while keeping data queryable in under a minute. See how other teams benchmark database migrations like this one by following backend and databases coverage on daily.dev.
How does Shopify handle constantly changing event schemas in ClickHouse without slowing down queries?
Shopify uses materialized views to track which fields exist across hundreds of daily-changing event shapes, powering autocomplete queries that return in 50-100 milliseconds. Hot keys stored in Map(String, String) columns get promoted to typed columns, running about 30% faster than the all-string baseline and compressing better, avoiding read-time type conversion. Developers wrestling with flexible schema design can track patterns like these on daily.dev.
Why would a company choose ClickHouse Cloud over self-hosting ClickHouse for observability workloads?
ClickHouse Cloud offers faster time-to-value than self-hosting by handling upgrades, scaling, and separation of storage and compute automatically. Separating ingest from query compute reduces the number of data replicas needed, cutting costs, while self-hosting requires teams to own every performance and cost trade-off, such as disk bandwidth, IOPS, and part-merge throughput, themselves. Weighing managed versus self-hosted database infrastructure is easier when comparisons like this surface on daily.dev.