--- title: "Day 28/30 AWS System Design Patterns" url: https://daily.dev/posts/day-28-30-aws-system-design-patterns-cmgv6kytm source_url: https://daily.dev/posts/day-28-30-aws-system-design-patterns-cmgv6kytm type: freeform source: "Joud Awad" author: "Joud Awad" published: 2026-08-26T17:12:56.536Z updated: 2026-08-26T17:13:24.213Z tags: ["aws", "career", "serverless", "distributed-systems", "aws-lambda"] reading_time: 2 upvotes: 17 comments: 7 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Day 28/30 AWS System Design Patterns **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 2 min read · 17 upvotes · 7 comments ## Summary A system design puzzle explains why webhooks fired 2-3 times when a third-party provider slowed to averaging 22 seconds per call. The Lambda function has a 30-second timeout matching the SQS visibility timeout exactly, and messages are delivered in batches of 5. The root cause: the visibility timeout starts when the event source mapping receives the message, not when processing begins, so later messages in a slow batch expire their visibility window before being processed and get redelivered to another invocation, causing duplicate webhook calls. ## Content A Lambda function _(serverless compute, 30-second timeout — exits after 30s regardless of whether processing is complete)_ processes SQS messages _(message queue — hides a delivered message for the visibility timeout, then re-delivers it if it hasn't been deleted)_ that trigger third-party webhook calls. The function has a 30-second timeout. Each webhook call to the external provider takes between 200 ms and 25 seconds depending on the provider's load. The SQS queue has a visibility timeout of 30 seconds — matching the Lambda timeout exactly. The event source mapping delivers messages in batches of 5. Under normal load the function processes 1,200 messages per hour without issue. During a period where the third-party provider is slow — averaging 22 seconds per call — support tickets start arriving. Customers report that their webhooks are firing 2–3 times instead of once. No code changes were made. The Lambda function has no explicit retry logic. What is causing duplicate webhook deliveries? A) SQS _(message queue, at-least-once delivery on Standard queues)_ is delivering duplicate messages — Standard queues occasionally send the same message twice independent of processing outcomes; but genuine SQS duplicates are rare, random, and not correlated with provider slowdowns B) The visibility timeout starts when the event source mapping receives the message — not when your code starts processing it; with 5 messages in a batch at 22 seconds each, the later messages in the batch are still waiting their turn when their 30-second visibility clock expires, and SQS re-delivers them to another invocation C) Lambda's SQS event source mapping _(polls SQS and invokes Lambda with batches of messages)_ automatically retries failed batches, sending the same message to multiple concurrent Lambda invocations; but the event source mapping processes different messages concurrently — it does not hand the same in-flight message to two invocations under normal operation D) The third-party provider is calling back to the platform confirming receipt, which triggers a second SQS message; but that would produce a new message with a new MessageId — not the same MessageId being processed twice Answer in the comments. #AWS #Serverless #SystemDesign #DistributedSystems #30DaysOfSystemDesign ## Community discussion Top comments from developers on daily.dev. **@joudawad** · 3 upvotes > The answer is B. > > The visibility timeout clock does not start when your code begins working on a message. It starts the moment the event source mapping calls `ReceiveMessage` and takes the batch off the queue. Everything after that point — invoke scheduling, and above all, the message's position in the batch — spends the clock before your code ever touches the message. > > Walk the timeline for a batch of 5 with the provider at 22 seconds per call: > > t=0 — the event source mapping receives all 5 messages. Five visibility clocks start simultaneously. > > t=0 to t=22 — message 1 is processed.... **@joudawad** · 1 upvotes > Also, it would mean a lot to me if you could support my content and stay in touch 🙏 > > - YouTube: [https://www.youtube.com/@system-design-lab](https://www.youtube.com/@system-design-lab) > - LinkedIn: [https://www.linkedin.com/in/joud-awad/](https://www.linkedin.com/in/joud-awad/) > - Medium Blog: [https://joudwawad.medium.com/](https://joudwawad.medium.com/) > - Substack: [https://joudawad.substack.com/](https://joudawad.substack.com/) **@joudawad** · 0 upvotes > A — Standard queues do produce genuine duplicates at a low, random rate. But random infrastructure duplicates don't correlate with a third-party provider's response times. Duplicates that appear exactly when the provider slows down and vanish when it recovers are a timing artifact, not queue behavior. **@joudawad** · 0 upvotes > C — The event source mapping *(polls SQS and invokes Lambda with batches)* runs multiple pollers, but each in-flight message belongs to exactly one batch and one invocation until its visibility timeout expires. The re-delivery in this incident happens because the timeout expired — which is option B, not a mapping defect. **@joudawad** · 0 upvotes > D — A provider callback would enqueue a new message with a new MessageId. Duplicate webhook firings traced to the same MessageId being processed by two invocations rule this out. --- Tags: [#aws](https://daily.dev/tags/aws), [#career](https://daily.dev/tags/career), [#serverless](https://daily.dev/tags/serverless), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#aws-lambda](https://daily.dev/tags/aws-lambda) [View this post on daily.dev](https://daily.dev/posts/day-28-30-aws-system-design-patterns-cmgv6kytm)