--- title: Event-Driven Architecture Failure Modes | daily.dev description: Event-driven architecture diagrams hide five critical assumptions that fail in production: exact-once delivery (SNS/SQS default to at-least-once, Kafka's... canonical: https://daily.dev/posts/event-driven-architecture-failure-modes-dex7gmqxe twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Event-Driven Architecture Failure Modes | daily.dev og:description: Event-driven architecture diagrams hide five critical assumptions that fail in production: exact-once delivery (SNS/SQS default to at-least-once, Kafka's... og:url: https://daily.dev/posts/event-driven-architecture-failure-modes-dex7gmqxe og:image: https://api.daily.dev/og/posts/DeX7gmqxe.png og:image:alt: Event-Driven Architecture Failure Modes og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Event-Driven Architecture Failure Modes **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 2 min read · 1 upvotes · 0 comments ## Summary Event-driven architecture diagrams hide five critical assumptions that fail in production: exact-once delivery (SNS/SQS default to at-least-once, Kafka's guarantee doesn't extend beyond Kafka-to-Kafka), ordered processing (SQS has no ordering, Kafka only within a partition), visible failures (logic errors get ACKed silently and surface days later), stable schemas (schema drift silently breaks downstream consumers), and built-in traceability (distributed tracing requires deliberate investment). The gap between a clean whiteboard diagram and reliable production operation is where teams get hurt. ## Content Your event-driven architecture diagram has 5 load-bearing assumptions hidden in it. None of them are labeled. All of them break in production. Every EDA whiteboard looks the same. Producer to broker to consumer. Clean arrows. Independent services. Then you deploy it. Events arrive out of order. A consumer crashes mid-processing. Nobody knows if the message was handled. A schema change silently breaks three downstream services. Your dead-letter queue fills up while dashboards show green. Here are the 5 assumptions your happy-path diagram doesn't show: 1. "Events will be delivered exactly once" They won't. SNS and SQS default to at-least-once. Kafka's exactly-once only covers Kafka-to-Kafka flows. The moment you write to PostgreSQL, you're back in duplicate territory. Your broker delivered the message. Your consumer debited the account twice. The broker calls that a success. 1. "Consumers will process events in order" SQS standard queues provide zero ordering guarantees. Kafka only orders within a single partition. Add retries, parallel consumers, or partition rebalancing, and even those guarantees collapse. An OrderShipped event processed before OrderConfirmed doesn't throw an error. It corrupts state silently. 1. "Failures will surface visibly" They don't. A consumer processes a malformed event, writes garbage to the database, and ACKs it. No exception. No alert. The damage shows up three days later in a customer support ticket. DLQs catch crashes. They don't catch logic errors. 1. "Event schemas won't change" They will. A producer adds a field, renames another, or changes a type. No contract. No versioning. Three downstream consumers break in three different ways. None of them know why. Schema drift is the silent killer of event-driven systems at scale. 1. "You can trace any event through the system" You can't. Not without deliberate investment. A correlation ID that doesn't propagate across every service boundary is useless. An event that passed through 4 services with no distributed trace is a black box. Observability in EDA isn't free. It's an engineering project in itself. The gap between "this works locally" and "this runs reliably at scale" is where teams get hurt. EDA works, but the diagram never shows you the cost of running it. --- Tags: [#architecture](https://daily.dev/tags/architecture), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#kafka](https://daily.dev/tags/kafka), [#observability](https://daily.dev/tags/observability) [View this post on daily.dev](https://daily.dev/posts/event-driven-architecture-failure-modes-dex7gmqxe) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/event-driven-architecture-failure-modes-dex7gmqxe","headline":"Event-Driven Architecture Failure Modes","text":"Event-driven architecture diagrams hide five critical assumptions that fail in production: exact-once delivery (SNS/SQS default to at-least-once, Kafka's guarantee doesn't extend beyond Kafka-to-Kafka), ordered processing (SQS has no ordering, Kafka only within a partition), visible failures (logic errors get ACKed silently and surface days later), stable schemas (schema drift silently breaks downstream consumers), and built-in traceability (distributed tracing requires deliberate investment). The gap between a clean whiteboard diagram and reliable production operation is where teams get hurt.","url":"https://daily.dev/posts/event-driven-architecture-failure-modes-dex7gmqxe","datePublished":"2026-03-23T20:53:08.104Z","dateModified":"2026-03-30T02:19:28.484Z","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0","description":"Principal Solution Architecture ","worksFor":{"@type":"Organization","name":"Metalab","logo":"https://www.google.com/s2/favicons?domain=metalab.com&sz=128"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":81460}},"image":"https://media.daily.dev/image/upload/s--AFwcZ-WC--/f_auto/v1774299198/posts/DeX7gmqxe?_a=BAMAMiWQ0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh","name":"Joud Awad"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Joud Awad","item":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh"},{"@type":"ListItem","position":3,"name":"Event-Driven Architecture Failure Modes"}]} ```