--- title: Day 3/30 AWS System Design Patterns | daily.dev description: A real-world AWS architecture puzzle: an e-commerce platform processed 8,000 orders/day without issues for 4 months. During a 40× traffic spike, 340 customers... canonical: https://daily.dev/posts/day-3-30-aws-system-design-patterns-rbxopelkr twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Day 3/30 AWS System Design Patterns | daily.dev og:description: A real-world AWS architecture puzzle: an e-commerce platform processed 8,000 orders/day without issues for 4 months. During a 40× traffic spike, 340 customers... og:url: https://daily.dev/posts/day-3-30-aws-system-design-patterns-rbxopelkr og:image: https://api.daily.dev/og/posts/rBxopeLKr.png og:image:alt: Day 3/30 AWS System Design Patterns og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Day 3/30 AWS System Design Patterns **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 2 min read · 57 upvotes · 25 comments ## Summary A real-world AWS architecture puzzle: an e-commerce platform processed 8,000 orders/day without issues for 4 months. During a 40× traffic spike, 340 customers were double-charged. CloudWatch showed EventBridge delivered each event exactly once, and Lambda had no custom retry logic. The culprit was Lambda's built-in async invocation retry behavior — when a DynamoDB write failed after a successful payment charge, Lambda retried the entire function, charging the card again. The payment function was not idempotent, so the retry caused a duplicate charge. The lesson: in distributed serverless systems, any operation that can be retried must be idempotent, especially side effects like payment charges. ## Content 340 customers were charged twice. No one touched the code. An e-commerce platform processes orders through a simple pipeline: EventBridge _(serverless event bus, invokes Lambda asynchronously — Lambda retries up to 2 times on any thrown error)_ receives an `order.placed` event, routes it to a Lambda _(serverless compute)_, which charges the payment and writes the confirmed order to DynamoDB _(managed NoSQL database)_. The architecture ran in production for 4 months — 8,000 orders per day, zero duplicate charges, zero incidents. The platform runs a promotional campaign. Traffic spikes 40× over 2 hours. 14,000 orders in one window. Post-campaign, the payments team pulls the report. 340 orders show two charges. CloudWatch _(AWS monitoring service)_ shows EventBridge delivered each event exactly once. The Lambda has no retry logic. No alerts fired during the campaign. The engineer on-call checks the Lambda logs. Every invocation shows a successful payment charge. Then — on 340 of them — a DynamoDB write error at the very end. Then a second invocation of the same function, for the same order, a few seconds later. The second invocation charges the card again. No code changed. No bug was introduced. The system behaved exactly as designed. What is the root cause? A) EventBridge _(serverless event bus, at-least-once delivery)_ delivered duplicate events during the spike — at-least-once delivery means duplicates increase under high throughput B) Lambda _(serverless compute, retries async invocations up to 2 times on any thrown error)_ retried failed invocations after the DynamoDB write threw — the payment charge ran again on retry because the function is not idempotent C) DynamoDB _(managed NoSQL database, can throttle writes under sudden traffic spikes)_ throttling caused a race condition between two concurrent Lambda invocations processing the same event D) The payment provider has its own retry logic — the platform's architecture is correct, the provider caused the double charge Answer in the comments. #SystemDesign #AWSLambda #Idempotency #DistributedSystems #ServerlessArchitecture ## Community discussion Top comments from developers on daily.dev. **@joudawad** · 7 upvotes > The answer is B. > > EventBridge _(serverless event bus)_ invokes Lambda _(serverless compute)_ asynchronously. When a Lambda function invoked asynchronously throws any error — for any reason — Lambda automatically retries the invocation up to 2 additional times. This is documented, default behavior. The retries are Lambda's responsibility, not EventBridge's. EventBridge did its job: it delivered the event once. > > Here is the exact sequence for each of the 340 duplicate charges: > > Lambda is invoked. It validates the order. It calls the payment provider — the charge succeeds. It then calls DynamoDB... **@joudawad** · 5 upvotes > Also, it would mean a lot to me if you could support my content and stay in touch 🙏 > > - YouTube: [https://www.youtube.com/@system-design-lab](https://www.youtube.com/@system-design-lab) > - LinkedIn: [https://www.linkedin.com/in/joud-awad/](https://www.linkedin.com/in/joud-awad/) > - Medium Blog: [https://joudwawad.medium.com/](https://joudwawad.medium.com/) > - Substack: [https://joudawad.substack.com/](https://joudawad.substack.com/) **@joudawad** · 3 upvotes > A — EventBridge does guarantee at-least-once delivery. But CloudWatch *(AWS monitoring service)* confirms each event was delivered exactly once. The pattern — successful payment, DynamoDB error, second invocation seconds later — is a Lambda retry signature, not an EventBridge duplicate. An EventBridge duplicate would produce two near-simultaneous invocations, not a sequential gap with an error in between. **@joudawad** · 3 upvotes > D — The payment provider received two separate, valid charge requests with no idempotency key. Honoring both is correct behavior on their part. The platform sent two requests. That is the platform's bug to fix. **@joudawad** · 2 upvotes > C — Lambda's async event source mapping invokes the function once per event and only schedules a retry after the first invocation completes with an error. The two invocations are separated by seconds and strictly sequential. No concurrency, no race. ## Similar posts on daily.dev - [Scaling a Financial Reconciliation Pipeline With Serverless](https://daily.dev/posts/scaling-a-financial-reconciliation-pipeline-with-serverless-gynvqfhrg) · Cloud Native Now · 0 upvotes · 0 comments --- Tags: [#distributed-systems](https://daily.dev/tags/distributed-systems), [#aws-lambda](https://daily.dev/tags/aws-lambda), [#aws-dynamodb](https://daily.dev/tags/aws-dynamodb) [View this post on daily.dev](https://daily.dev/posts/day-3-30-aws-system-design-patterns-rbxopelkr) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/day-3-30-aws-system-design-patterns-rbxopelkr","headline":"Day 3/30 AWS System Design Patterns","text":"A real-world AWS architecture puzzle: an e-commerce platform processed 8,000 orders/day without issues for 4 months. During a 40× traffic spike, 340 customers were double-charged. CloudWatch showed EventBridge delivered each event exactly once, and Lambda had no custom retry logic. The culprit was Lambda's built-in async invocation retry behavior — when a DynamoDB write failed after a successful payment charge, Lambda retried the entire function, charging the card again. The payment function was not idempotent, so the retry caused a duplicate charge. The lesson: in distributed serverless systems, any operation that can be retried must be idempotent, especially side effects like payment charges.","url":"https://daily.dev/posts/day-3-30-aws-system-design-patterns-rbxopelkr","datePublished":"2026-07-23T14:34:08.274Z","dateModified":"2026-09-13T21:23:15.770Z","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0","description":"Principal Solution Architecture ","worksFor":{"@type":"Organization","name":"Metalab","logo":"https://www.google.com/s2/favicons?domain=metalab.com&sz=128"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":86200}},"image":"https://media.daily.dev/image/upload/s--AoAYPBnT--/f_auto/v1784817267/posts/rBxopeLKr?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":57},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":25}],"comment":[{"@type":"Comment","text":"The answer is B.\nEventBridge (serverless event bus) invokes Lambda (serverless compute) asynchronously. When a Lambda function invoked asynchronously throws any error — for any reason — Lambda automatically retries the invocation up to 2 additional times. This is documented, default behavior. The retries are Lambda’s responsibility, not EventBridge’s. EventBridge did its job: it delivered the event once.\nHere is the exact sequence for each of the 340 duplicate charges:\nLambda is invoked. It validates the order. It calls the payment provider — the charge succeeds. It then calls DynamoDB (managed NoSQL database) to write the confirmed order. Under the 40× traffic spike, DynamoDB is throttling writes and returns a ProvisionedThroughputExceededException. Lambda sees a thrown error, marks the invocation as failed, and schedules a retry.\nOn retry, Lambda runs the full function from the beginning. It calls the payment provider again. The payment provider has no record of a previous charge — the function passed no idempotency key. The card is charged a second time. The DynamoDB write succeeds on retry. The order appears in the database once. The card was charged twice.\nTwo fixes:\nOption 1 — Pass a stable idempotency key to the payment provider on every charge request. Use order_id. If the provider receives a second charge request with the same key, it returns the result of the original charge without processing a new one.\nOption 2 — Before calling the payment provider, check DynamoDB for an existing payment record keyed on order_id. If one exists, skip the charge.\nLambda retries are a feature, not a failure. Every Lambda function invoked asynchronously must be safe to run multiple times.","datePublished":"2026-07-23T14:35:49.175Z","url":"https://daily.dev/posts/rBxopeLKr#c-HvvfCPIYv","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":7}},{"@type":"Comment","text":"Also, it would mean a lot to me if you could support my content and stay in touch 🙏\n\nYouTube: https://www.youtube.com/@system-design-lab\nLinkedIn: https://www.linkedin.com/in/joud-awad/\nMedium Blog: https://joudwawad.medium.com/\nSubstack: https://joudawad.substack.com/","datePublished":"2026-07-23T14:35:52.315Z","url":"https://daily.dev/posts/rBxopeLKr#c-ZVnIZeDnw","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":5}},{"@type":"Comment","text":"A — EventBridge does guarantee at-least-once delivery. But CloudWatch (AWS monitoring service) confirms each event was delivered exactly once. The pattern — successful payment, DynamoDB error, second invocation seconds later — is a Lambda retry signature, not an EventBridge duplicate. An EventBridge duplicate would produce two near-simultaneous invocations, not a sequential gap with an error in between.","datePublished":"2026-07-23T14:34:39.677Z","url":"https://daily.dev/posts/rBxopeLKr#c-TeciHvz7R","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3}},{"@type":"Comment","text":"D — The payment provider received two separate, valid charge requests with no idempotency key. Honoring both is correct behavior on their part. The platform sent two requests. That is the platform’s bug to fix.","datePublished":"2026-07-23T14:34:53.756Z","url":"https://daily.dev/posts/rBxopeLKr#c-HUSKsZTEz","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3}},{"@type":"Comment","text":"C — Lambda’s async event source mapping invokes the function once per event and only schedules a retry after the first invocation completes with an error. The two invocations are separated by seconds and strictly sequential. No concurrency, no race.","datePublished":"2026-07-23T14:34:46.923Z","url":"https://daily.dev/posts/rBxopeLKr#c-PgBY26iwA","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh","name":"Joud Awad"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Joud Awad","item":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh"},{"@type":"ListItem","position":3,"name":"Day 3/30 AWS System Design Patterns"}]} ```