--- title: 41/60 Days System Design Questions | daily.dev description: A system design challenge asking readers to choose the right real-time streaming technology to meet a 500ms fraud detection SLA, given 8,000 events/sec on AWS.... canonical: https://daily.dev/posts/41-60-days-system-design-questions-bqy6smteh twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: 41/60 Days System Design Questions | daily.dev og:description: A system design challenge asking readers to choose the right real-time streaming technology to meet a 500ms fraud detection SLA, given 8,000 events/sec on AWS.... og:url: https://daily.dev/posts/41-60-days-system-design-questions-bqy6smteh og:image: https://api.daily.dev/og/posts/bqY6sMtEH.png og:image:alt: 41/60 Days System Design Questions og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # 41/60 Days System Design Questions **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 1 min read · 125 upvotes · 21 comments ## Summary A system design challenge asking readers to choose the right real-time streaming technology to meet a 500ms fraud detection SLA, given 8,000 events/sec on AWS. The four options are Kafka Streams, Apache Flink, Spark Structured Streaming, and keeping the batch job with a 1-minute window. The post hints that three of the four options can meet the latency requirement, and invites readers to pick one and justify their choice. ## Content Your data team just got a new SLA: surface fraud signals within 500ms of a transaction. Right now you're running nightly Spark batch jobs. The business wants "real-time." Your team knows Spark. Someone already opened a PR adding Spark Structured Streaming. The transaction volume: 8,000 events/sec peak. You're on AWS. The fraud model runs in Python. The output feeds a DynamoDB table the API reads from. You need to redesign the pipeline. What do you pick? **A)** Kafka Streams — event-by-event processing, stateful operators, sub-10ms latency. Lives inside your app JVM. **B)** Apache Flink — true streaming engine, exactly-once semantics, built for high-throughput stateful processing. **C)** Spark Structured Streaming — micro-batch under the hood, 100ms–5s windows, same API your team already knows. **D)** Keep the batch job, drop the window to 1 minute — "near real-time" at zero migration cost. Three of these can hit sub-500ms. One of them cannot — no matter how you tune it. Pick one — A, B, C, or D — and tell me why. Full breakdown in the comments. Drop your answer 👇 #30DaysOfSystemDesign #SystemDesign #DataEngineering #DistributedSystems ## Community discussion Top comments from developers on daily.dev. **@joudawad** · 9 upvotes > **Why B wins (Apache Flink):** > > Flink is a true streaming engine — event-by-event, not batch. Every transaction triggers processing the moment it arrives. Latency: sub-50ms end-to-end with proper tuning. At 8K events/sec, Flink doesn't blink. Exactly-once semantics mean your fraud signals aren't double-counted when a node fails. The Python fraud model integrates via PyFlink or a sidecar microservice. Flink's watermark model handles out-of-order events cleanly — which matters when transactions arrive from distributed payment processors with clock skew. > > > Operational overhead is real. But the... **@joudawad** · 9 upvotes > **Why A is close but not the cleanest fit (Kafka Streams):** > > Kafka Streams is event-by-event too — sub-10ms achievable. The constraint: it runs inside your application JVM. Your fraud model is Python. That means Kafka Streams handles the stream topology, but the model scoring lives in a separate Python service, adding a network hop. Flink handles that boundary more cleanly with PyFlink. Kafka Streams is the right call if the fraud logic is stateful JVM-native aggregation. Wrong default when the model is a Python service. **@joudawad** · 4 upvotes > **Why C is the senior engineer trap (Spark Structured Streaming):** > > The PR is already open. The API looks familiar. But Structured Streaming is micro-batch — it collects events for a trigger interval (practically 100ms–5s under production load), then processes them as a mini-batch. You'll tune it, it'll pass staging, and it'll breach SLA the first time peak traffic hits. Mistaking "Streaming" in the name for actual event-by-event processing is the trap. **@joudawad** · 4 upvotes > **Why D is eliminated immediately (batch, 1-minute windows):** > > Minimum latency = the window size. At 60 seconds you're 120x over SLA before the first line of fraud logic runs. "Near real-time" is not a SLA. **@joudawad** · 4 upvotes > Also, it would mean a lot to me if you could support my content and stay in touch 🙏 > > - YouTube: [https://www.youtube.com/@system-design-lab](https://www.youtube.com/@system-design-lab) > - LinkedIn: [https://www.linkedin.com/in/joud-awad/](https://www.linkedin.com/in/joud-awad/) > - Medium Blog: [https://joudwawad.medium.com/](https://joudwawad.medium.com/) > - Substack: [https://joudawad.substack.com/](https://joudawad.substack.com/) ## Similar posts on daily.dev - [Why Real-Time Stream Processing Beats Batch ETL for AI Data Freshness in 2026](https://daily.dev/posts/why-real-time-stream-processing-beats-batch-etl-for-ai-data-freshness-in-2026-c1hypylls) · Confluent Blog · 3 upvotes · 0 comments --- Tags: [#career](https://daily.dev/tags/career), [#apache-spark](https://daily.dev/tags/apache-spark), [#apache-kafka](https://daily.dev/tags/apache-kafka), [#apache-flink](https://daily.dev/tags/apache-flink) [View this post on daily.dev](https://daily.dev/posts/41-60-days-system-design-questions-bqy6smteh) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/41-60-days-system-design-questions-bqy6smteh","headline":"41/60 Days System Design Questions","text":"A system design challenge asking readers to choose the right real-time streaming technology to meet a 500ms fraud detection SLA, given 8,000 events/sec on AWS. The four options are Kafka Streams, Apache Flink, Spark Structured Streaming, and keeping the batch job with a 1-minute window. The post hints that three of the four options can meet the latency requirement, and invites readers to pick one and justify their choice.","url":"https://daily.dev/posts/41-60-days-system-design-questions-bqy6smteh","datePublished":"2026-06-16T16:37:19.023Z","dateModified":"2026-06-16T16:37:40.545Z","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0","description":"Principal Solution Architecture ","worksFor":{"@type":"Organization","name":"Metalab","logo":"https://www.google.com/s2/favicons?domain=metalab.com&sz=128"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":82050}},"image":"https://media.daily.dev/image/upload/s--c1HNex4t--/f_auto/v1781627844/posts/bqY6sMtEH?_a=BAMAMiWQ0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":125},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":21}],"comment":[{"@type":"Comment","text":"Why B wins (Apache Flink):\nFlink is a true streaming engine — event-by-event, not batch. Every transaction triggers processing the moment it arrives. Latency: sub-50ms end-to-end with proper tuning. At 8K events/sec, Flink doesn’t blink. Exactly-once semantics mean your fraud signals aren’t double-counted when a node fails. The Python fraud model integrates via PyFlink or a sidecar microservice. Flink’s watermark model handles out-of-order events cleanly — which matters when transactions arrive from distributed payment processors with clock skew.\nOperational overhead is real. But the SLA demands it. You’re not reaching for Flink prematurely here — the business asked for 500ms and gave you 8K events/sec. That’s exactly the problem Flink was built for.","datePublished":"2026-06-16T16:37:50.261Z","url":"https://daily.dev/posts/bqY6sMtEH#c-leAOKkp5p","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":9}},{"@type":"Comment","text":"Why A is close but not the cleanest fit (Kafka Streams):\nKafka Streams is event-by-event too — sub-10ms achievable. The constraint: it runs inside your application JVM. Your fraud model is Python. That means Kafka Streams handles the stream topology, but the model scoring lives in a separate Python service, adding a network hop. Flink handles that boundary more cleanly with PyFlink. Kafka Streams is the right call if the fraud logic is stateful JVM-native aggregation. Wrong default when the model is a Python service.","datePublished":"2026-06-16T16:37:53.111Z","url":"https://daily.dev/posts/bqY6sMtEH#c-B1GX7qn8y","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":9}},{"@type":"Comment","text":"Why C is the senior engineer trap (Spark Structured Streaming):\nThe PR is already open. The API looks familiar. But Structured Streaming is micro-batch — it collects events for a trigger interval (practically 100ms–5s under production load), then processes them as a mini-batch. You’ll tune it, it’ll pass staging, and it’ll breach SLA the first time peak traffic hits. Mistaking “Streaming” in the name for actual event-by-event processing is the trap.","datePublished":"2026-06-16T16:37:57.600Z","url":"https://daily.dev/posts/bqY6sMtEH#c-jhLC1oOr0","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":4}},{"@type":"Comment","text":"Why D is eliminated immediately (batch, 1-minute windows):\nMinimum latency = the window size. At 60 seconds you’re 120x over SLA before the first line of fraud logic runs. “Near real-time” is not a SLA.","datePublished":"2026-06-16T16:38:01.487Z","url":"https://daily.dev/posts/bqY6sMtEH#c-rDeb9RBIX","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":4}},{"@type":"Comment","text":"Also, it would mean a lot to me if you could support my content and stay in touch 🙏\n\nYouTube: https://www.youtube.com/@system-design-lab\nLinkedIn: https://www.linkedin.com/in/joud-awad/\nMedium Blog: https://joudwawad.medium.com/\nSubstack: https://joudawad.substack.com/","datePublished":"2026-06-16T16:38:17.614Z","url":"https://daily.dev/posts/bqY6sMtEH#c-ew3zeJteX","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":4}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh","name":"Joud Awad"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Joud Awad","item":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh"},{"@type":"ListItem","position":3,"name":"41/60 Days System Design Questions"}]} ```