--- title: Razorpay cuts warehouse refresh 10x, incident.io halves... description: Razorpay replaced a full-refresh Spark pipeline with an incremental graph-and-index approach, cutting compute time 90% across their 50+ denormalized Facts.... canonical: https://daily.dev/posts/razorpay-cuts-warehouse-refresh-10x-incident-io-halves-primary-cpu-with-read-replica-h3nltpogb twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Razorpay cuts warehouse refresh 10x, incident.io halves primary CPU with read replica | daily.dev og:description: Razorpay replaced a full-refresh Spark pipeline with an incremental graph-and-index approach, cutting compute time 90% across their 50+ denormalized Facts.... og:url: https://daily.dev/posts/razorpay-cuts-warehouse-refresh-10x-incident-io-halves-primary-cpu-with-read-replica-h3nltpogb og:image: https://api.daily.dev/og/posts/h3nLTpogb.png og:image:alt: Razorpay cuts warehouse refresh 10x, incident.io halves primary CPU with read replica og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Razorpay cuts warehouse refresh 10x, incident.io halves primary CPU with read replica **[Databases Digest](https://daily.dev/sources/databases_digest)** · 5 min read · 0 upvotes · 0 comments ## Summary Razorpay replaced a full-refresh Spark pipeline with an incremental graph-and-index approach, cutting compute time 90% across their 50+ denormalized Facts. Incident.io documented a clean LSN-based read-after-write solution that moved 60%+ of reads to a replica and halved primary CPU. Bolt migrated 10 million monthly Looker queries from Presto to Databricks, saving seven figures annually, with the key technical win being a custom Liquid-aware SQLGlot extension. Redis patched three confirmed zero-days from the Kimi K3 vulnerability report in releases 8.8.1 and corresponding older branch updates. ## Content **TLDR:** Razorpay replaced a full-refresh Spark pipeline with an incremental graph-and-index approach, cutting compute time 90% across their 50+ denormalized Facts. Incident.io documented a clean LSN-based read-after-write solution that moved 60%+ of reads to a replica and halved primary CPU. Bolt migrated 10 million monthly Looker queries from Presto to Databricks, saving seven figures annually, with the key technical win being a custom Liquid-aware SQLGlot extension. Redis patched three confirmed zero-days from the Kimi K3 vulnerability report in releases 8.8.1 and corresponding older branch updates. --- ## Razorpay's incremental warehouse refresh: dependency graphs and secondary indexes cut runtime 90% Razorpay modeled their Fact configs as traversable dependency graphs to determine exactly which rows need updating on any change event, eliminating the full-refresh Spark/EMR runs that were hammering their 50+ denormalized Facts. They built lightweight secondary indexes per entity on the data lake to remove full table scans and cut the TiDB warm-store dependency for historical lookups. Iceberg V2 with merchant_id bucketing handles high-cardinality dimension joins at runtime instead of precomputing everything. The five largest Facts migrated first; pipeline runtime dropped ~90% with full historical coverage restored. [Read more](https://daily.dev/posts/13upK8Os9) ## Incident.io moves 60%+ of reads to a PostgreSQL replica using LSN stamping and context-based taint propagation The core read-after-write problem — routing a read to the replica before the write has replicated — is solved by propagating a taint through Go context whenever a mutation occurs, then routing subsequent reads to primary within that request. Cross-boundary consistency (message queue workers consuming events from a different process) uses PostgreSQL LSN stamping on published messages, so workers wait until the replica has caught up to that LSN before reading. The same LSN technique applies to HTTP middleware. Result: 60%+ of reads now hit the replica, primary CPU cut in half, and replica lag acts as natural back pressure rather than a source of outages. [Read more](https://daily.dev/posts/Upell649q) ## Bolt migrates 10M monthly Looker queries from Presto to Databricks with a custom SQLGlot Liquid tokenizer The migration achieved 45% faster queries and 40% lower costs — seven figures in annual savings. The hard part was not the SQL dialect gap but Looker's Liquid templating embedded in LookML: regex placeholders failed, LLM-based translation failed, and the breakthrough was extending SQLGlot with a Liquid-aware tokenizer, parser, and generator that treats Liquid as a proper grammar rather than noise. Validation used dashboard-level correctness checks comparing Presto vs. Databricks outputs. The team's honest lesson: budget heavily for edge cases in the final 10% of any migration. [Read more](https://daily.dev/posts/DMc6IM0fG) ## Redis patches three confirmed zero-days in 8.8.1 after Kimi K3 vulnerability report Researchers used the Kimi K3 AI model to claim 19 zero-days in Redis. Redis confirmed three: a Streams use-after-free, a RedisBloom TDigest out-of-bounds write, and a RedisBloom TopK RDB loader wild-free. All three had already been reported independently via the bug bounty program. Redis 8.8.1 and six older branch updates are out; the TopK open source fix is still pending. If you run RedisBloom in any form, update now and verify network isolation. [Read more](https://daily.dev/posts/PVfy7A9c4) --- ## Also notable - **LINE app runs E2EE on Kafka at 1M messages/second with under 1% CPU overhead:** LY Corporation's design uses AES-GCM DEKs encrypted with ECC secp521r1/ECIES KEKs, a shared KEK model to keep header size stable regardless of consumer count, and a plaintext fallback for zero-downtime rollout — full production at 1M msg/s with CPU overhead below 1% per instance. [Read more](https://daily.dev/posts/SCal5iZBa) - **Spotify's Random Access Parquet reduces point queries on the data lake to a single ranged read:** An external index maps lookup keys to exact Parquet file offsets and row numbers, and write-time optimizations (sorting, one-page-per-key, ZSTD frame resets, column interleaving) can shrink a point query to a few kilobytes — letting the same files serve batch analytics and low-latency online lookups without ETL duplication. [Read more](https://daily.dev/posts/y3p7o9Ihw) - **PostgreSQL MVCC compared across Oracle, InnoDB, SQL Server, MongoDB, CockroachDB, and etcd:** A thorough breakdown of the four design questions every MVCC implementation must answer, showing that the costs are conserved — each engine only chooses who pays and how it fails — and that Postgres's choice makes garbage visible and maintenance manual but never cancels a reader with 'snapshot too old'. [Read more](https://daily.dev/posts/0Puk6JB2F) - **Fujitsu contributed 83 commits to PostgreSQL 19, including logical replication of sequences and update_deleted conflict detection:** Key additions: table exclusions in FOR ALL TABLES publications, ALTER SUBSCRIPTION REFRESH SEQUENCES for logical replication of sequences, and update_deleted conflict detection as a step toward multi-master replication. [Read more](https://daily.dev/posts/uXk3u0A20) - **Azure DocumentDB query tuned from 333K document scan to 18 index keys in 0.053ms using ESR compound indexing:** The guide walks through reading explain() output and applying the Equality-Sort-Range rule to compound index design, with covered queries eliminating FETCH stages entirely — a concrete before/after that's directly applicable to any document store with similar index mechanics. [Read more](https://daily.dev/posts/q5aVUBS2P) --- Tags: [#security](https://daily.dev/tags/security), [#backend](https://daily.dev/tags/backend), [#postgresql](https://daily.dev/tags/postgresql), [#data-engineering](https://daily.dev/tags/data-engineering), [#redis](https://daily.dev/tags/redis) [View this post on daily.dev](https://daily.dev/posts/razorpay-cuts-warehouse-refresh-10x-incident-io-halves-primary-cpu-with-read-replica-h3nltpogb) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/razorpay-cuts-warehouse-refresh-10x-incident-io-halves-primary-cpu-with-read-replica-h3nltpogb","headline":"Razorpay cuts warehouse refresh 10x, incident.io halves primary CPU with read replica","text":"Razorpay replaced a full-refresh Spark pipeline with an incremental graph-and-index approach, cutting compute time 90% across their 50+ denormalized Facts. Incident.io documented a clean LSN-based read-after-write solution that moved 60%+ of reads to a replica and halved primary CPU. Bolt migrated 10 million monthly Looker queries from Presto to Databricks, saving seven figures annually, with the key technical win being a custom Liquid-aware SQLGlot extension. Redis patched three confirmed zero-days from the Kimi K3 vulnerability report in releases 8.8.1 and corresponding older branch updates.","url":"https://daily.dev/posts/razorpay-cuts-warehouse-refresh-10x-incident-io-halves-primary-cpu-with-read-replica-h3nltpogb","datePublished":"2026-07-28T04:18:00.697Z","dateModified":"2026-07-28T04:18:25.446Z","author":{"@type":"Organization","name":"Databases Digest","logo":"https://media.daily.dev/image/upload/s--ir1hdV8b--/f_auto,q_auto/v1773839400/logos/databases_digest?_a=BAMAMiiu0","url":"https://daily.dev/sources/databases_digest"},"interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/databases_digest","name":"Databases Digest"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Databases Digest","item":"https://daily.dev/sources/databases_digest"},{"@type":"ListItem","position":3,"name":"Razorpay cuts warehouse refresh 10x, incident.io halves primary CPU with read replica"}]} ```