--- title: Day 10/30 AWS System Design Patterns | daily.dev description: A Lambda-based order service writes to DynamoDB then publishes to EventBridge. Over 8 months, 217 orders were silently lost — paid and stored, but never heard... canonical: https://daily.dev/posts/day-10-30-aws-system-design-patterns-yiz9wpjz7 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Day 10/30 AWS System Design Patterns | daily.dev og:description: A Lambda-based order service writes to DynamoDB then publishes to EventBridge. Over 8 months, 217 orders were silently lost — paid and stored, but never heard... og:url: https://daily.dev/posts/day-10-30-aws-system-design-patterns-yiz9wpjz7 og:image: https://api.daily.dev/og/posts/yIz9wpJZ7.png og:image:alt: Day 10/30 AWS System Design Patterns og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Day 10/30 AWS System Design Patterns **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 3 min read · 77 upvotes · 13 comments ## Summary A Lambda-based order service writes to DynamoDB then publishes to EventBridge. Over 8 months, 217 orders were silently lost — paid and stored, but never heard by downstream services — because process crashes between the two writes left no error to catch. The post walks through four candidate fixes (reorder writes, add retries, use transactions, use DynamoDB Streams) and explains why only one is correct: stop publishing from application code and instead use DynamoDB Streams to trigger a publisher Lambda from the committed record, eliminating the dual-write race condition entirely. ## Content An order service runs on Lambda _(serverless compute)_. When a customer checks out, the handler does two things, in order: write the order to DynamoDB _(managed NoSQL database)_, then publish an `order.placed` event to EventBridge _(serverless event bus — routes events to fraud, email, and warehouse services)_. Two calls, four lines apart. The pattern has run for 8 months at 12,000 orders per day. A quarterly reconciliation job compares the orders table against the warehouse system. It finds 217 orders over the past 3 weeks that exist in DynamoDB — paid, confirmed, stored — but that no downstream service ever heard about. No fraud score, no confirmation email, no shipment. Support tickets confirm it: "I paid and nothing happened." The team searches the logs. There is no error for any of the 217. What they do find: each one lines up with a deployment, a function timeout, or an out-of-memory kill. The handler wrote the order, and then the process died before line four. The engineers also find the mirror image, once: an `order.placed` event for an order that does not exist in the table — from an experiment months ago when someone published first "to be safe." What is the correct fix? A) Publish to EventBridge first, then write to DynamoDB — if the publish succeeds, downstream always hears about the order; but reversing two non-atomic writes just reverses the failure: now a crash after the publish creates ghost events for orders that were never saved, which is the mirror-image bug they already found once B) Wrap the publish in retry logic with exponential backoff — retries help when a call fails and the process lives to see the failure; a timeout or crash between the two calls kills the retry loop along with everything else, and the 217 losses left no failure for any code to react to C) Stop publishing from application code — make the DynamoDB write the only write, and derive the event from the table's change stream: DynamoDB Streams _(ordered change log of every write to the table, retained 24 hours)_ triggers a small publisher Lambda that emits `order.placed` from the committed record D) Use DynamoDB transactions (`TransactWriteItems`) to make the order write and the EventBridge publish atomic — but a DynamoDB transaction coordinates writes across DynamoDB items only; no database transaction can enlist an external message bus, so atomicity ends at the storage boundary Answer in the comments. **#SystemDesign #AWS #Serverless #DistributedSystems #EventDrivenArchitecture** ## Community discussion Top comments from developers on daily.dev. **@joudawad** · 5 upvotes > The answer is C. > > This is the dual-write problem, and it is a property of the design, not a bug in the code. The handler performs two writes into two systems that share no transaction: DynamoDB _(managed NoSQL database)_ and EventBridge _(serverless event bus)_. Between those two calls there is a window — milliseconds wide — where the process can die: a timeout, a deploy, an OOM kill, a runtime crash. Nothing that dies inside that window leaves an error behind, because the error handler died too. A milliseconds-wide window multiplied by 12,000 orders a day, every day, is not a risk. It is a... **@fahimahammed** · 3 upvotes > The correct answer is **C**. > > Here's why: > > - **A is incorrect:** Reversing the order doesn't solve the problem—it just creates the opposite inconsistency. If the event is published first and the process crashes before writing to DynamoDB, downstream services process an order that doesn't exist (a "ghost event"). > - **B is incorrect:** Retries only help when the application is still running and receives an error. In this scenario, the Lambda function was terminated (timeout, OOM, deployment) after the database write but before the publish. There's no opportunity for retry logic to execute. > -... **@joudawad** · 2 upvotes > B — Retry code is part of the process, and the failure mode here is the process dying. The 217 lost orders produced no exception, no failed response, no signal that any retry loop — however well-tuned — could have reacted to. Retries fix failed calls; they cannot fix vanished callers. **@joudawad** · 2 upvotes > A — There is no ordering of two non-atomic writes that makes them atomic. Write-then-publish loses events; publish-then-write manufactures them. The team has already observed both bugs in production — they were choosing which inconsistency to prefer, not whether to have one. **@tokstesla** · 2 upvotes > It's always fun learning from your posts. ## Similar posts on daily.dev - [How to Lose Data Using AWS Serverless Without Even Knowing It](https://daily.dev/posts/how-to-lose-data-using-aws-serverless-without-even-knowing-it-4ra8u44mm) · Serverless Life · 1 upvotes · 0 comments - [5 Backend System Design Patterns Every Engineer Should Actually Understand](https://daily.dev/posts/5-backend-system-design-patterns-every-engineer-should-actually-understand-nzc6ercbu) · Medium · 5 upvotes · 0 comments - [How to Fix the Dual-Write Problem in Node.js with the Outbox Pattern](https://daily.dev/posts/how-to-fix-the-dual-write-problem-in-node-js-with-the-outbox-pattern-kobnx54pc) · freeCodeCamp · 1 upvotes · 0 comments - [Scaling a Financial Reconciliation Pipeline With Serverless](https://daily.dev/posts/scaling-a-financial-reconciliation-pipeline-with-serverless-gynvqfhrg) · Cloud Native Now · 0 upvotes · 0 comments --- Tags: [#architecture](https://daily.dev/tags/architecture), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#aws-lambda](https://daily.dev/tags/aws-lambda), [#aws-dynamodb](https://daily.dev/tags/aws-dynamodb) [View this post on daily.dev](https://daily.dev/posts/day-10-30-aws-system-design-patterns-yiz9wpjz7) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/day-10-30-aws-system-design-patterns-yiz9wpjz7","headline":"Day 10/30 AWS System Design Patterns","text":"A Lambda-based order service writes to DynamoDB then publishes to EventBridge. Over 8 months, 217 orders were silently lost — paid and stored, but never heard by downstream services — because process crashes between the two writes left no error to catch. The post walks through four candidate fixes (reorder writes, add retries, use transactions, use DynamoDB Streams) and explains why only one is correct: stop publishing from application code and instead use DynamoDB Streams to trigger a publisher Lambda from the committed record, eliminating the dual-write race condition entirely.","url":"https://daily.dev/posts/day-10-30-aws-system-design-patterns-yiz9wpjz7","datePublished":"2026-07-30T22:23:22.173Z","dateModified":"2026-09-13T21:34:17.845Z","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0","description":"Principal Solution Architecture ","worksFor":{"@type":"Organization","name":"Metalab","logo":"https://www.google.com/s2/favicons?domain=metalab.com&sz=128"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":84960}},"image":"https://media.daily.dev/image/upload/s--lCDqhV8R--/f_auto/v1785450221/posts/yIz9wpJZ7?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":77},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":13}],"comment":[{"@type":"Comment","text":"The answer is C.\nThis is the dual-write problem, and it is a property of the design, not a bug in the code. The handler performs two writes into two systems that share no transaction: DynamoDB (managed NoSQL database) and EventBridge (serverless event bus). Between those two calls there is a window — milliseconds wide — where the process can die: a timeout, a deploy, an OOM kill, a runtime crash. Nothing that dies inside that window leaves an error behind, because the error handler died too. A milliseconds-wide window multiplied by 12,000 orders a day, every day, is not a risk. It is a schedule. The 217 divergences were guaranteed; only their order numbers were random.\nThe fix is to stop having two writes. The application performs exactly one: the order goes into DynamoDB. That write either commits or it doesn’t — one atomic fact. DynamoDB Streams (ordered change log of committed writes, retained 24 hours) then carries that fact forward: an event source mapping invokes a publisher Lambda (serverless compute) for each stream record, and the publisher emits order.placed. The event is no longer a thing the application promises to do after saving. It is derived from the save itself. If the order committed, the event will follow — the stream and its retries see to it. If the order never committed, no event can exist. Both failure modes from the scenario become unrepresentable.\nThis is the transactional outbox pattern, with the outbox table you’d build by hand on a relational database replaced by the change log DynamoDB already keeps. In the wider world the same move is called change data capture — Debezium tailing MySQL’s binlog, Postgres logical replication feeding Kafka. One sentence to keep: if it didn’t come from the commit log, it’s a rumor.\nThe trade-offs, honestly: the stream is at-least-once, so the publisher can emit a duplicate order.placed — downstream consumers need idempotency on order_id (day 3 and day 8 readers already know this tax; it is the same tax everywhere). Events now trail the write by a short, usually sub-second delay. And the publisher is real infrastructure: give its event source mapping a bounded retry and an on-failure destination, because the one component whose job is completeness must not be allowed to fail silently.","datePublished":"2026-07-30T22:24:22.922Z","url":"https://daily.dev/posts/yIz9wpJZ7#c-g8yIMMrl8","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":5}},{"@type":"Comment","text":"The correct answer is C.\nHere’s why:\n\nA is incorrect: Reversing the order doesn’t solve the problem—it just creates the opposite inconsistency. If the event is published first and the process crashes before writing to DynamoDB, downstream services process an order that doesn’t exist (a “ghost event”).\nB is incorrect: Retries only help when the application is still running and receives an error. In this scenario, the Lambda function was terminated (timeout, OOM, deployment) after the database write but before the publish. There’s no opportunity for retry logic to execute.\n✅ C is correct: This is the Transactional Outbox pattern (implemented using DynamoDB Streams).The application performs one durable write: the order to Amazon DynamoDB.DynamoDB Streams captures every committed write.A separate Lambda triggered by the stream publishes the order.placed event to Amazon EventBridge.Since the event is derived from the committed database change, the “write succeeded but event was lost” window disappears.\nD is incorrect: DynamoDB transactions (TransactWriteItems) only guarantee atomicity for DynamoDB items. They cannot include external systems like Amazon EventBridge in the same transaction.","datePublished":"2026-07-31T14:33:20.451Z","url":"https://daily.dev/posts/yIz9wpJZ7#c-mlOEQ34dn","author":{"@type":"Person","name":"Fahim Ahammed Firoz","url":"https://daily.dev/fahimahammed","image":"https://lh3.googleusercontent.com/a/ACg8ocLPDnjeV4QsaVwi1lXMBgMAKbog9I4fgMTTJnHRKE-Uc1NmSJY0=s96-c"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3}},{"@type":"Comment","text":"B — Retry code is part of the process, and the failure mode here is the process dying. The 217 lost orders produced no exception, no failed response, no signal that any retry loop — however well-tuned — could have reacted to. Retries fix failed calls; they cannot fix vanished callers.","datePublished":"2026-07-30T22:24:37.685Z","url":"https://daily.dev/posts/yIz9wpJZ7#c-wKb3sRvEP","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"A — There is no ordering of two non-atomic writes that makes them atomic. Write-then-publish loses events; publish-then-write manufactures them. The team has already observed both bugs in production — they were choosing which inconsistency to prefer, not whether to have one.","datePublished":"2026-07-30T22:24:32.274Z","url":"https://daily.dev/posts/yIz9wpJZ7#c-B0Nxtxslo","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"It’s always fun learning from your posts.","datePublished":"2026-07-31T14:31:24.085Z","url":"https://daily.dev/posts/yIz9wpJZ7#c-piQjOJ92K","author":{"@type":"Person","name":"Tochukwu Okeh","url":"https://daily.dev/tokstesla","image":"https://lh3.googleusercontent.com/a/ALm5wu0M9zC8KvFRBWLakb25xsuWq6eewxwbl6yGQ8Gp=s96-c"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh","name":"Joud Awad"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Joud Awad","item":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh"},{"@type":"ListItem","position":3,"name":"Day 10/30 AWS System Design Patterns"}]} ```