--- title: Day 2/30 AWS System Design Patterns | daily.dev description: A fintech payment platform case study exposes a critical AWS EventBridge pitfall: when the analytics team disables their rule during a 4-hour maintenance... canonical: https://daily.dev/posts/day-2-30-aws-system-design-patterns-rywf9jvwd twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Day 2/30 AWS System Design Patterns | daily.dev og:description: A fintech payment platform case study exposes a critical AWS EventBridge pitfall: when the analytics team disables their rule during a 4-hour maintenance... og:url: https://daily.dev/posts/day-2-30-aws-system-design-patterns-rywf9jvwd og:image: https://api.daily.dev/og/posts/ryWf9jVWD.png og:image:alt: Day 2/30 AWS System Design Patterns og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Day 2/30 AWS System Design Patterns **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 3 min read · 115 upvotes · 24 comments ## Summary A fintech payment platform case study exposes a critical AWS EventBridge pitfall: when the analytics team disables their rule during a 4-hour maintenance window, EventBridge silently discards 14,000 events with no storage or retry, causing $1.4M in missing reconciliation data. The scenario presents four architectural alternatives — DLQ on the rule, SQS as the rule target, Kinesis Data Streams with consumer checkpointing, or EventBridge Archive and Replay — and asks readers to identify the correct fix for a durable, replayable event consumer. ## Content A fintech startup is building a real-time payment processing platform. Three teams need to react when a payment completes: the notifications team (send push + email), the fraud team (score the transaction), and the analytics team (write to the data warehouse for reporting and reconciliation — including re-processing historical events whenever finance restates a report). The backend architect picks EventBridge _(serverless event bus — rules are evaluated at publish time; an event that matches no enabled rule is discarded, not stored)_ for all three. Each team wires up a rule that triggers their Lambda _(serverless compute)_. Three teams, three rules, one bus. Clean. Six months later, the data warehouse goes down for a planned 4-hour migration on Sunday night. To stop their Lambda from throwing 14,000 errors against a dead warehouse, the analytics team disables their EventBridge rule for the window. Standard procedure. They re-enable it Monday at 6 AM. 14,000 payment events fire during those 4 hours. The fraud and notifications rules process every one of them. For the analytics rule, EventBridge evaluates each event, finds no enabled match, and moves on. No delivery is attempted. Nothing fails. Nothing is retried. Nothing is stored. $1.4M in transactions are missing from the Monday reconciliation report. The audit team flags it by 9 AM. The architect chose EventBridge for all three consumers. That was the mistake. What should the analytics team have been wired to instead? A) Enable a DLQ _(Dead Letter Queue — captures events EventBridge attempted to deliver to a target but could not)_ on the analytics rule — Sunday's 14,000 events would have been captured and re-driven after the maintenance window B) Use SQS _(message queue — buffers messages until a consumer processes and deletes them; each message is consumed once, then gone)_ as the rule target — the queue absorbs events while the pipeline is down, and the Lambda drains it on resume without disabling anything C) Consume from a Kinesis Data Stream _(streaming log — records are retained 24 hours to 7 days regardless of whether anyone reads them; each consumer owns its read position and resumes from it)_ — during maintenance the pipeline simply stops, then resumes from its last checkpoint and reads everything it missed D) Enable EventBridge Archive + Replay _(stores events published to the bus; an operator can later replay them, scoped to selected rules)_ — the archive retains Sunday's events and a Monday-morning replay re-delivers them to the analytics rule Answer in the comments. #SystemDesign #AWS #Serverless #CloudArchitecture #SoftwareEngineering ## Community discussion Top comments from developers on daily.dev. **@joudawad** · 11 upvotes > The answer is C. > > EventBridge _(serverless event bus)_ is a router, not a store. Rules are evaluated at the moment an event is published. An event that matches an enabled rule is delivered to that rule's targets; an event that matches nothing is discarded on the spot. A disabled rule is nothing. So for those 4 hours there was no delivery attempt, no error, no retry, and no DLQ entry — because from EventBridge's perspective, nothing went wrong. The system did exactly what it was configured to do. That is the defining property of a router: if no one is listed to receive the event right now, the... **@joudawad** · 6 upvotes > For a deep dive into AWS SNS vs SQS vs EventBridge please refer to my youtube video: > [https://www.youtube.com/watch?v=HMjo4Xy73b4](https://www.youtube.com/watch?v=HMjo4Xy73b4) **@joudawad** · 4 upvotes > Also, it would mean a lot to me if you could support my content and stay in touch 🙏 > > - YouTube: [https://www.youtube.com/@system-design-lab](https://www.youtube.com/@system-design-lab) > - LinkedIn: [https://www.linkedin.com/in/joud-awad/](https://www.linkedin.com/in/joud-awad/) > - Medium Blog: [https://joudwawad.medium.com/](https://joudwawad.medium.com/) > - Substack: [https://joudawad.substack.com/](https://joudawad.substack.com/) **@jad85** · 3 upvotes > Really love how you are delivering the pattern explanation rather than the AWS services them self in your explanation a great start for this series! really! **@joudawad** · 3 upvotes > B — Honest credit first: had the team targeted SQS *(message queue — consume-once)* and simply left everything running, the queue would have absorbed Sunday's events. Buffering while a consumer is offline is exactly what queues are for, and this incident alone does not rule SQS out. The stated requirement does: reconciliation and restatements mean re-reading events that were already successfully processed. SQS deletes on consume — there is nothing left to re-read. Buffering is not replay. --- Tags: [#aws](https://daily.dev/tags/aws), [#career](https://daily.dev/tags/career), [#architecture](https://daily.dev/tags/architecture), [#serverless](https://daily.dev/tags/serverless) [View this post on daily.dev](https://daily.dev/posts/day-2-30-aws-system-design-patterns-rywf9jvwd) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/day-2-30-aws-system-design-patterns-rywf9jvwd","headline":"Day 2/30 AWS System Design Patterns","text":"A fintech payment platform case study exposes a critical AWS EventBridge pitfall: when the analytics team disables their rule during a 4-hour maintenance window, EventBridge silently discards 14,000 events with no storage or retry, causing $1.4M in missing reconciliation data. The scenario presents four architectural alternatives — DLQ on the rule, SQS as the rule target, Kinesis Data Streams with consumer checkpointing, or EventBridge Archive and Replay — and asks readers to identify the correct fix for a durable, replayable event consumer.","url":"https://daily.dev/posts/day-2-30-aws-system-design-patterns-rywf9jvwd","datePublished":"2026-07-22T15:29:35.178Z","dateModified":"2026-07-23T04:17:31.162Z","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0","description":"Principal Solution Architecture ","worksFor":{"@type":"Organization","name":"Metalab","logo":"https://www.google.com/s2/favicons?domain=metalab.com&sz=128"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":81930}},"image":"https://media.daily.dev/image/upload/s--68N231iY--/f_auto/v1784734185/posts/ryWf9jVWD?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":115},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":24}],"comment":[{"@type":"Comment","text":"The answer is C.\nEventBridge (serverless event bus) is a router, not a store. Rules are evaluated at the moment an event is published. An event that matches an enabled rule is delivered to that rule’s targets; an event that matches nothing is discarded on the spot. A disabled rule is nothing. So for those 4 hours there was no delivery attempt, no error, no retry, and no DLQ entry — because from EventBridge’s perspective, nothing went wrong. The system did exactly what it was configured to do. That is the defining property of a router: if no one is listed to receive the event right now, the event ceases to exist.\nA Kinesis Data Stream (streaming log — retains records independently of consumption) inverts the model. The producer writes a record; the record is stored for the retention window whether zero consumers or ten consumers are running. Each consumer tracks its own position in the stream — a checkpoint. Warehouse maintenance now requires disabling nothing: the analytics consumer stops, records accumulate in the stream, and on Monday it resumes from its checkpoint and processes the backlog. A finance restatement is the same motion — rewind the checkpoint and re-read.\nThe trade-offs are real: retention is finite (24 hours by default, 7 days extended, 365 days at additional cost — backfills beyond that need an S3 archive), you manage shard capacity, and the payment events must be produced into the stream — either the payment service writes to it directly, or a permanent EventBridge rule targets the stream. Note that this rule is one no team ever has a reason to disable: the stream absorbs regardless of consumer state.\nThe pattern: match each consumer’s guarantee to the primitive. Notifications needs push delivery and tolerates loss — an EventBridge rule is right. Fraud needs delivery plus failure capture — EventBridge plus a Lambda failure destination. Analytics needs retention and re-consumption — that is a stream, and only a stream. This triad — bus (routes, no retention), queue (buffers, consume-once), log (retains, replayable) — is not an AWS quirk. It is the same distinction you make choosing between RabbitMQ, SQS, and Kafka.","datePublished":"2026-07-22T15:30:28.076Z","url":"https://daily.dev/posts/ryWf9jVWD#c-znJgSD2xT","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":11}},{"@type":"Comment","text":"For a deep dive into AWS SNS vs SQS vs EventBridge please refer to my youtube video:\nhttps://www.youtube.com/watch?v=HMjo4Xy73b4","datePublished":"2026-07-22T15:31:55.425Z","url":"https://daily.dev/posts/ryWf9jVWD#c-SgmIZCW1h","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":6}},{"@type":"Comment","text":"Also, it would mean a lot to me if you could support my content and stay in touch 🙏\n\nYouTube: https://www.youtube.com/@system-design-lab\nLinkedIn: https://www.linkedin.com/in/joud-awad/\nMedium Blog: https://joudwawad.medium.com/\nSubstack: https://joudawad.substack.com/","datePublished":"2026-07-22T15:30:51.718Z","url":"https://daily.dev/posts/ryWf9jVWD#c-e4ZrRKynJ","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":4}},{"@type":"Comment","text":"Really love how you are delivering the pattern explanation rather than the AWS services them self in your explanation a great start for this series! really!","datePublished":"2026-07-22T15:40:50.627Z","url":"https://daily.dev/posts/ryWf9jVWD#c-ojFIqwUbc","author":{"@type":"Person","name":"Jad","url":"https://daily.dev/jad85"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3}},{"@type":"Comment","text":"B — Honest credit first: had the team targeted SQS (message queue — consume-once) and simply left everything running, the queue would have absorbed Sunday’s events. Buffering while a consumer is offline is exactly what queues are for, and this incident alone does not rule SQS out. The stated requirement does: reconciliation and restatements mean re-reading events that were already successfully processed. SQS deletes on consume — there is nothing left to re-read. Buffering is not replay.","datePublished":"2026-07-22T15:30:39.680Z","url":"https://daily.dev/posts/ryWf9jVWD#c-tLqzEHSMW","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh","name":"Joud Awad"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Joud Awad","item":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh"},{"@type":"ListItem","position":3,"name":"Day 2/30 AWS System Design Patterns"}]} ```