Stripe just sent your server a charge.succeeded webhook.
Your pod was mid-restart. The webhook timed out. Stripe will retry — but now you’re racing.
Here’s the setup:
NestJS API on ECS, 4 tasks behind an ALB. Stripe POSTs charge.succeeded events to /webhooks/stripe. You have ~10s to return a 200 before Stripe times out and queues a retry. On Black Friday you’re handling ~80 webhooks/sec.
Three failure modes are already happening in prod:
• Pod restarts mid-deploy → webhook missed → Stripe retries → you process it, but a second pod also processed the original → double charge confirmation email sent
• DB is slow at peak → handler takes 12s → Stripe times out → retries → now you’re processing the same event twice while the first one is still running
• Stripe sends charge.succeeded then charge.refunded 200ms apart → your queue delivers them out of order → you mark the order paid AFTER it was already refunded
The handler is a 30-line function that does the right thing 99% of the time. The 1% is killing you.
What do you do?
A) Idempotency key + dedup table — store every event ID on receipt, skip if already seen.
B) Retry with exponential backoff + dead-letter queue — never drop a webhook, process at-least-once.
C) Verify signature, return 200 immediately, push to an internal queue — decouple receipt from processing.
D) Sequence numbers + reorder buffer — hold events until the prior sequence has landed, then process in strict order.
Three of these are patterns senior engineers genuinely debate in design reviews. Only one of them actually solves the problem in front of you.
Pick one — A, B, C, or D — and tell me why. Full breakdown in the comments.
If your team has ever shipped a webhook handler and prayed, share this with them. The debate is worth more than the post.
Drop your answer 👇
#30DaysOfSystemDesign #Day11 #SystemDesign #Webhooks

Smiley Face

Joud Awad

A system design challenge focused on handling Stripe webhook reliability at scale. The scenario presents a NestJS API on ECS with 4 tasks behind an ALB processing ~80 webhooks/sec on Black Friday. Three real production failure modes are described: double processing during pod restarts, duplicate processing due to slow DB causing Stripe retries, and out-of-order event delivery. Readers are asked to choose between four architectural patterns: idempotency key with dedup table, retry with exponential backoff and DLQ, immediate 200 response with internal queue decoupling, or sequence numbers with reorder buffer.

11/30 Days System Design Questions!

Answer: C — Verify signature, return 200 immediately, push to an internal queue ✅
Here’s why, and why the other three trick smart engineers:
Why C wins (Sync receipt + async processing):
The fundamental problem is that Stripe’s clock and your handler’s clock are decoupled. Stripe gives you ~10 seconds to ACK. Your DB write, your fraud check, your email send — none of that should live inside that 10-second window. Ever.
The pattern: validate the HMAC signature in &lt;5ms, write the raw payload to SQS / Kafka / Redis Streams, return 200. Now your handler is bounded by network + signature verification, not by whatever your downstream services are doing today. Pod restart mid-processing? The message is already durable in the queue — a worker picks it up. DB slow? The webhook endpoint doesn’t care — it returned 200 in 8ms. This is what Stripe, Shopify, and GitHub literally tell you to do in their own webhook docs.
<img src="https://media.daily.dev/image/upload/s--fx23EuZm--/f_auto/v1778949677/ugc/content_4f44fc37-5b68-4c74-a9d8-8c8394e1d4e9?_a=BAMAMiWQ0" alt="ChatGPT Image May 16, 2026, 07_37_08 PM (1).png">

System design isn’t just for clearing FAANG interviews. it’s the blueprint for building resilient software. The shift from writing functional code to thinking about data flow, load balancing, and database trade-offs is what turns a coder into an architect. Saving this guide for the architectural breakdown. Solid share!

Also, it would mean a lot to me if you could support my content and stay in touch 🙏
<ul>
<li>YouTube: <a href="https://www.youtube.com/@system-design-lab" target="_blank" rel="noopener nofollow">https://www.youtube.com/@system-design-lab</a></li>
<li>LinkedIn: <a href="https://www.linkedin.com/in/joud-awad/" target="_blank" rel="noopener nofollow">https://www.linkedin.com/in/joud-awad/</a></li>
<li>Medium Blog: <a href="https://joudwawad.medium.com/" target="_blank" rel="noopener nofollow">https://joudwawad.medium.com/</a></li>
<li>Substack: <a href="https://joudawad.substack.com/" target="_blank" rel="noopener nofollow">https://joudawad.substack.com/</a></li>
</ul>

Why D is wrong (Sequence numbers + reorder buffer):
This sounds rigorous and it’s the answer engineers reach for after they’ve been burned by ordering once. The problem: most webhook providers don’t give you sequence numbers. Stripe doesn’t. GitHub doesn’t. Even when they do (Shopify’s X-Shopify-Webhook-Id), the gaps aren’t dense — you can’t tell “missing event 47” from “47 was filtered.”
More importantly: ordering at the receiver is the wrong layer. You handle ordering at the domain level — when you process charge.refunded, you check the current state of the order before transitioning. If the order is already in refunded, you no-op. State machines beat reorder buffers because they handle the case where events arrive out of order AND the case where one of them never arrives at all. A reorder buffer just means you stall forever when an event gets dropped upstream.

Why B is wrong as the answer (Retry + DLQ):
Stripe is already doing the retry with exponential backoff. That’s their job, not yours. Stripe retries failed webhooks for up to 3 days with backoff. You don’t need to build that for inbound webhooks — you’d be duplicating provider behavior.
Retry + DLQ is the right pattern for your internal queue (the one C creates), not for the webhook endpoint itself. If your worker fails to process a queued event, sure, retry it and DLQ after N attempts. But that’s a downstream concern. Putting “retry logic” at the webhook receiver level confuses inbound and outbound retry semantics.