--- title: Shopify powers observability for global-scale commerce... description: Shopify replaced a fragmented, unpredictable multi-vendor observability stack with an internal platform called Observe, built entirely on self-hosted... canonical: https://daily.dev/posts/shopify-powers-observability-for-global-scale-commerce-with-clickhouse-r45dmqcus twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Shopify powers observability for global-scale commerce with ClickHouse | daily.dev og:description: Shopify replaced a fragmented, unpredictable multi-vendor observability stack with an internal platform called Observe, built entirely on self-hosted... og:url: https://daily.dev/posts/shopify-powers-observability-for-global-scale-commerce-with-clickhouse-r45dmqcus og:image: https://api.daily.dev/og/posts/r45DMqcUS.png og:image:alt: Shopify powers observability for global-scale commerce with ClickHouse og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Shopify powers observability for global-scale commerce with ClickHouse **[ClickHouse](https://daily.dev/sources/clickhouse)** · 9 min read · 3 upvotes · 0 comments ## Summary Shopify replaced a fragmented, unpredictable multi-vendor observability stack with an internal platform called Observe, built entirely on self-hosted ClickHouse. The migration delivered a 16x query improvement out of the box (peaking past 30x) while ingesting around 100 million events per second (~110 GB/s) at Black Friday-Cyber Monday peak, keeping data queryable in under a minute. Engineering director Elijah McPherson details techniques used at this scale: buffering telemetry through Kafka before durable batch commits, using materialized views for fast autocomplete and cross-signal correlation, promoting hot Map(String,String) keys to typed columns for ~30% faster queries, sorting map keys for 20-40% disk savings, and running an in-house Kubernetes operator to manage schema migrations across ~20 tenants. Looking ahead, Shopify is moving to tiered storage, ARM-based compute, ClickHouse's native JSON type, and a Rust-based ingestion pipeline, while also exploring an AI agent layer and merchant-facing analytics. McPherson notes that if starting today, Shopify would likely choose ClickHouse Cloud instead of self-hosting to offload operational burden. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How many events per second does Shopify ingest into its observability platform during Black Friday Cyber Monday peak? Shopify ingests around 100 million events per second at Black Friday-Cyber Monday peak, translating to roughly 110 GB/s of uncompressed telemetry, compared to about 50 million events per second at steady state. The platform keeps this data queryable in under a minute across metrics, logs, traces, exceptions, and profiles, all built on ClickHouse. _Teams sizing observability pipelines for peak commerce traffic can find real-world benchmarks like this through daily.dev._ ### How does Shopify keep ClickHouse queries fast when hundreds of teams emit constantly changing event schemas? Shopify relies on materialized views: a metadata view tracks which fields exist and their values so autocomplete queries return in 50-100 milliseconds, while hot keys get promoted out of Map(String,String) into typed columns, running about 30% faster than an all-string baseline and compressing better. A separate materialized view also correlates identifiers across datasets, giving roughly a 10x speedup on cross-signal lookups. _Engineers wrestling with flexible schemas at scale can track patterns like this on daily.dev._ ### Why would Shopify choose ClickHouse Cloud instead of self-hosted ClickHouse if starting over today? Shopify's engineering director says he would likely choose ClickHouse Cloud because it offers faster time-to-value with separated storage and compute, automatic upgrades, and scaling handled out of the box, reducing the operational burden of self-hosting. Separating ingest from query compute also means needing fewer data replicas, which lowers cost compared to their current self-managed setup. _Teams weighing self-hosted versus managed ClickHouse can follow real operator tradeoffs on daily.dev._ ## Similar posts on daily.dev - [How the D. E. Shaw group powers high-cardinality observability at scale with ClickHouse](https://daily.dev/posts/how-the-d-e-shaw-group-powers-high-cardinality-observability-at-scale-with-clickhouse-51pw0hsby) · ClickHouse · 0 upvotes · 0 comments - [How Trigger.dev is using ClickHouse to scale observability for long-running AI workflows](https://daily.dev/posts/how-trigger-dev-is-using-clickhouse-to-scale-observability-for-long-running-ai-workflows-1urnkfdl2) · ClickHouse · 0 upvotes · 0 comments - [Have you heard? Clickhouse is winning the observability wars\!](https://daily.dev/posts/have-you-heard-clickhouse-is-winning-the-observability-wars--5bpefin3l) · Charity · 18 upvotes · 0 comments - [ClickHouse achieves AWS Cloud Operations Competency](https://daily.dev/posts/clickhouse-achieves-aws-cloud-operations-competency-8uvx6eyrc) · ClickHouse · 1 upvotes · 0 comments --- Tags: [#observability](https://daily.dev/tags/observability), [#shopify](https://daily.dev/tags/shopify), [#clickhouse](https://daily.dev/tags/clickhouse) [View this post on daily.dev](https://daily.dev/posts/shopify-powers-observability-for-global-scale-commerce-with-clickhouse-r45dmqcus) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Shopify powers observability for global-scale commerce with ClickHouse","url":"https://daily.dev/posts/shopify-powers-observability-for-global-scale-commerce-with-clickhouse-r45dmqcus","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/shopify-powers-observability-for-global-scale-commerce-with-clickhouse-r45dmqcus"},"datePublished":"2026-08-20T10:22:36.563Z","dateModified":"2026-09-13T21:34:55.968Z","description":"Shopify replaced a fragmented, unpredictable multi-vendor observability stack with an internal platform called Observe, built entirely on self-hosted...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/febce9dd9d92c9521827230920fa7745?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/febce9dd9d92c9521827230920fa7745?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"ClickHouse","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ClickHouse","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/clickhouse","url":"https://daily.dev/sources/clickhouse"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/shopify-powers-observability-for-global-scale-commerce-with-clickhouse-r45dmqcus","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"observability,shopify,clickhouse","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ClickHouse","item":"https://daily.dev/sources/clickhouse"},{"@type":"ListItem","position":3,"name":"Shopify powers observability for global-scale commerce with ClickHouse"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/shopify-powers-observability-for-global-scale-commerce-with-clickhouse-r45dmqcus#faq","mainEntity":[{"@type":"Question","name":"How many events per second does Shopify ingest into its observability platform during Black Friday Cyber Monday peak?","acceptedAnswer":{"@type":"Answer","text":"Shopify ingests around 100 million events per second at Black Friday-Cyber Monday peak, translating to roughly 110 GB/s of uncompressed telemetry, compared to about 50 million events per second at steady state. The platform keeps this data queryable in under a minute across metrics, logs, traces, exceptions, and profiles, all built on ClickHouse. Teams sizing observability pipelines for peak commerce traffic can find real-world benchmarks like this through daily.dev."}},{"@type":"Question","name":"How does Shopify keep ClickHouse queries fast when hundreds of teams emit constantly changing event schemas?","acceptedAnswer":{"@type":"Answer","text":"Shopify relies on materialized views: a metadata view tracks which fields exist and their values so autocomplete queries return in 50-100 milliseconds, while hot keys get promoted out of Map(String,String) into typed columns, running about 30% faster than an all-string baseline and compressing better. A separate materialized view also correlates identifiers across datasets, giving roughly a 10x speedup on cross-signal lookups. Engineers wrestling with flexible schemas at scale can track patterns like this on daily.dev."}},{"@type":"Question","name":"Why would Shopify choose ClickHouse Cloud instead of self-hosted ClickHouse if starting over today?","acceptedAnswer":{"@type":"Answer","text":"Shopify's engineering director says he would likely choose ClickHouse Cloud because it offers faster time-to-value with separated storage and compute, automatic upgrades, and scaling handled out of the box, reducing the operational burden of self-hosting. Separating ingest from query compute also means needing fewer data replicas, which lowers cost compared to their current self-managed setup. Teams weighing self-hosted versus managed ClickHouse can follow real operator tradeoffs on daily.dev."}}]} ```