A Lambda function (serverless compute, 30-second timeout — exits after 30s regardless of whether processing is complete) processes SQS messages (message queue — hides a delivered message for the visibility timeout, then re-delivers it if it hasn’t been deleted) that trigger third-party webhook calls. The function has a 30-second timeout. Each webhook call to the external provider takes between 200 ms and 25 seconds depending on the provider’s load. The SQS queue has a visibility timeout of 30 seconds — matching the Lambda timeout exactly. The event source mapping delivers messages in batches of 5.
Under normal load the function processes 1,200 messages per hour without issue. During a period where the third-party provider is slow — averaging 22 seconds per call — support tickets start arriving. Customers report that their webhooks are firing 2–3 times instead of once.
No code changes were made. The Lambda function has no explicit retry logic.
What is causing duplicate webhook deliveries?
A) SQS (message queue, at-least-once delivery on Standard queues) is delivering duplicate messages — Standard queues occasionally send the same message twice independent of processing outcomes; but genuine SQS duplicates are rare, random, and not correlated with provider slowdowns
B) The visibility timeout starts when the event source mapping receives the message — not when your code starts processing it; with 5 messages in a batch at 22 seconds each, the later messages in the batch are still waiting their turn when their 30-second visibility clock expires, and SQS re-delivers them to another invocation
C) Lambda’s SQS event source mapping (polls SQS and invokes Lambda with batches of messages) automatically retries failed batches, sending the same message to multiple concurrent Lambda invocations; but the event source mapping processes different messages concurrently — it does not hand the same in-flight message to two invocations under normal operation
D) The third-party provider is calling back to the platform confirming receipt, which triggers a second SQS message; but that would produce a new message with a new MessageId — not the same MessageId being processed twice
Answer in the comments.
#AWS #Serverless #SystemDesign #DistributedSystems #30DaysOfSystemDesign

Joud Awad

A system design puzzle explains why webhooks fired 2-3 times when a third-party provider slowed to averaging 22 seconds per call. The Lambda function has a 30-second timeout matching the SQS visibility timeout exactly, and messages are delivered in batches of 5. The root cause: the visibility timeout starts when the event source mapping receives the message, not when processing begins, so later messages in a slow batch expire their visibility window before being processed and get redelivered to another invocation, causing duplicate webhook calls.

Day 28/30 AWS System Design Patterns

The answer is B.
The visibility timeout clock does not start when your code begins working on a message. It starts the moment the event source mapping calls <code>ReceiveMessage</code> and takes the batch off the queue. Everything after that point — invoke scheduling, and above all, the message’s position in the batch — spends the clock before your code ever touches the message.
Walk the timeline for a batch of 5 with the provider at 22 seconds per call:
t=0 — the event source mapping receives all 5 messages. Five visibility clocks start simultaneously.
t=0 to t=22 — message 1 is processed. Fine.
t=22 to t=30 — message 2 is mid-processing when its visibility timeout expires at t=30. Messages 3, 4, and 5 haven’t been touched at all — and their clocks just expired too.
t=30 onward — SQS (message queue) marks messages 2 through 5 visible again and delivers them to the next poll. A second invocation starts calling the same webhooks while the first invocation is still working through its batch. The first invocation eventually finishes and the batch delete fails quietly for the re-delivered messages — but the duplicate webhooks have already fired.
Nothing errored. Every invocation succeeded. The duplication comes purely from the clock starting at receive time while processing time is paid per message, in sequence.
Two fixes:
Fix 1 — Set the visibility timeout to at least 6 times the function timeout. This is AWS’s documented guidance for SQS event source mappings, and the batch math above is exactly why: the visibility window must cover the worst case of (batch size × per-message processing time) plus scheduling overhead, not just one execution. For a 30-second function timeout, that is a 180-second visibility timeout.
Fix 2 — Make the webhook call idempotent anyway. Standard queues are at-least-once by contract; even with a correct visibility timeout, occasional re-delivery is guaranteed behavior, not a bug. Send an idempotency key (the MessageId works) so the receiving side can deduplicate.
The deeper lesson: visibility timeout is not “how long my code runs.” It is “how long SQS keeps this message exclusively mine, measured from receipt.” Those are different quantities, and batches drive them apart.
<img src="https://media.daily.dev/image/upload/s--m9pN1GJS--/f_auto/v1787764417/ugc/content_01934b09-86ea-420d-9afd-ac5502ca27eb?_a=BAMAMicg0" alt="ChatGPT Image Aug 26, 2026, 08_13_23 PM.png">

Also, it would mean a lot to me if you could support my content and stay in touch 🙏
<ul>
<li>YouTube: <a href="https://www.youtube.com/@system-design-lab" target="_blank" rel="noopener nofollow">https://www.youtube.com/@system-design-lab</a></li>
<li>LinkedIn: <a href="https://www.linkedin.com/in/joud-awad/" target="_blank" rel="noopener nofollow">https://www.linkedin.com/in/joud-awad/</a></li>
<li>Medium Blog: <a href="https://joudwawad.medium.com/" target="_blank" rel="noopener nofollow">https://joudwawad.medium.com/</a></li>
<li>Substack: <a href="https://joudawad.substack.com/" target="_blank" rel="noopener nofollow">https://joudawad.substack.com/</a></li>
</ul>

A — Standard queues do produce genuine duplicates at a low, random rate. But random infrastructure duplicates don’t correlate with a third-party provider’s response times. Duplicates that appear exactly when the provider slows down and vanish when it recovers are a timing artifact, not queue behavior.

C — The event source mapping (polls SQS and invokes Lambda with batches) runs multiple pollers, but each in-flight message belongs to exactly one batch and one invocation until its visibility timeout expires. The re-delivery in this incident happens because the timeout expired — which is option B, not a mapping defect.

D — A provider callback would enqueue a new message with a new MessageId. Duplicate webhook firings traced to the same MessageId being processed by two invocations rule this out.