Holding Revamp Went Live. Then Reality Check Hit Hard
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
A post-mortem from Groww Engineering covering three production bugs discovered after rolling out a Holdings service revamp serving millions of users. First, a race condition between Hibernate's @PostUpdate callback and transaction commit caused stale ETag responses — fixed by deferring cache invalidation to after transaction commit using @TransactionalEventListener. Second, a 1.5s CDC pipeline delay (caused by per-message schema verification in CockroachDB Changefeed) was misread as Kafka queue latency — fixed with schema_locked=true, reducing propagation latency by 81%. Third, database CPU hit 72% due to non-covering secondary indexes forcing double lookups — fixed with a covering index, dropping CPU to ~16% at full rollout despite 2× index storage cost.
Table of contents
IntroductionChapter 1: The Ghost ETagThe CulpritThe FixLessonChapter 2: The One-Second Delay We Couldn’t AffordThe CulpritGet Jainnirmal ’s stories in your inboxThe FixLessonChapter 3: The Database That Kept Asking for More CPUThe CulpritThe FixLessonKey TakeawaysQuestions this post answers
Why does @PostUpdate in Hibernate fire before the transaction commits and how can I safely invalidate a cache after commit?
@PostUpdate executes after Hibernate flushes entity changes but before the surrounding transaction commits. This creates a race window where another request can read stale data and generate a fresh ETag before the commit is visible. The safe fix is to publish a domain event from @PostUpdate and process it in a @TransactionalEventListener with phase = TransactionPhase.AFTER_COMMIT, ensuring cache invalidation only happens once all changes are committed and visible to other transactions. Teams shipping cache invalidation logic around JPA lifecycle hooks track these transaction boundary pitfalls on daily.dev.
How do I reduce CockroachDB Changefeed latency that is adding ~1 second delay before messages reach Kafka?
CockroachDB Changefeed performs schema verification before every publication by default. When the table schema is stable, this repeated verification adds significant overhead — roughly 1 second per message in one production case. Running ALTER TABLE table_name SET (schema_locked = true) skips repeated verification and can reduce end-to-end CDC propagation latency by around 81%, from ~1.54s to ~281ms. Engineers tuning CDC pipelines between CockroachDB and Kafka share findings like this on daily.dev.
How does a covering index reduce database CPU usage compared to a regular secondary index?
A non-covering secondary index locates matching rows but then requires a second lookup against the primary index to fetch columns not stored in the index — effectively two physical reads per logical query. A covering index stores all required response columns alongside the index keys, serving queries entirely from the index. In one high-traffic production case, switching to a covering index dropped peak database CPU from ~72% to ~16% at full rollout, at the cost of approximately 2× index storage. Developers optimizing read-heavy query plans at scale find real production numbers like these on daily.dev.
59.8K Impressions2 Comments