--- title: Day 27/30 AWS System Design Patterns | daily.dev description: A system design scenario walks through architecting an IoT data pipeline that must support both sub-500ms anomaly detection and cold storage archival to S3 as... canonical: https://daily.dev/posts/day-27-30-aws-system-design-patterns-zshlhvrvq twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Day 27/30 AWS System Design Patterns | daily.dev og:description: A system design scenario walks through architecting an IoT data pipeline that must support both sub-500ms anomaly detection and cold storage archival to S3 as... og:url: https://daily.dev/posts/day-27-30-aws-system-design-patterns-zshlhvrvq og:image: https://api.daily.dev/og/posts/ZshlHvRvq.png og:image:alt: Day 27/30 AWS System Design Patterns og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Day 27/30 AWS System Design Patterns **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 2 min read · 104 upvotes · 25 comments ## Summary A system design scenario walks through architecting an IoT data pipeline that must support both sub-500ms anomaly detection and cold storage archival to S3 as Parquet files. It compares Kinesis Data Streams and Kinesis Data Firehose, explaining why Firehose has no Lambda destination, why writing individual S3 objects from Streams creates small-file problems, and why direct Lambda invocation from devices lacks buffering. The correct design uses Kinesis Data Streams as the ingestion layer with two consumers: the anomaly detector reading directly from the stream, and a Firehose delivery stream subscribed as a second consumer to handle batching and S3 archival. ## Content A streaming data platform ingests IoT sensor readings from 50,000 devices. Each device sends a reading every 10 seconds. Readings are 512 bytes each. Total ingestion rate: 5,000 readings per second, approximately 2.5 MB/s. The data team has two requirements. First: a real-time anomaly detection service that needs to process each reading within 500 ms of ingestion. Second: a cold storage archive where all readings land in S3 _(object storage, optimized for large batch writes not individual records)_ as Parquet files for quarterly analysis, with up to 5 minutes of delivery latency acceptable. The engineering team is debating whether to use Kinesis Data Streams _(real-time stream, sub-second consumer latency, pull model)_ or Kinesis Data Firehose _(managed delivery pipe — batches records to S3/Redshift/OpenSearch with buffering configurable from 0 to 900 seconds; Lambda is not an available destination)_. Which architecture is correct? A) Use Kinesis Data Firehose for both — configure two delivery streams, one targeting the anomaly detector Lambda and one targeting S3; but Firehose has no Lambda destination at all — Lambda exists only as an inline transform inside the delivery pipeline, so no Firehose configuration hands each record to your detector B) Use Kinesis Data Streams for both — configure two consumers, one for real-time processing and one writing to S3 directly; but writing one S3 object per record or small batch creates millions of tiny files that destroy query performance and recreate Firehose's batching logic poorly C) Use Kinesis Data Streams _(real-time stream, sub-second latency)_ as the ingestion layer — the anomaly detector consumes directly from the stream within milliseconds, and a Firehose delivery stream _(managed delivery pipe, handles batching + Parquet conversion automatically)_ subscribes to the Data Stream as a second consumer for S3 archival D) Publish directly from devices to the anomaly detector Lambda _(serverless compute)_ and use Firehose separately for S3 — but 50,000 devices at 5,000 requests/second means 5,000 concurrent Lambda invocations with no buffer, no backpressure, and no retention if the detector falls behind Answer in the comments. #SystemDesign #AWS #DataEngineering #StreamingData #CloudArchitecture ## Community discussion Top comments from developers on daily.dev. **@joudawad** · 10 upvotes > The answer is C. > > Kinesis Data Streams _(real-time stream, sub-second consumer latency)_ and Kinesis Data Firehose _(managed delivery pipe, buffers to S3/Redshift/OpenSearch)_ solve different problems and are designed to work together. > > Kinesis Data Streams is a real-time streaming service. Consumers read from it with sub-second latency using shard iterators or enhanced fan-out. It retains records for 24 hours to 7 days and supports multiple independent consumers, each maintaining their own read position. This is the right tool for the anomaly detection service that needs records within 500... **@paoloap** · 2 upvotes > C, because Firehose's minimum S3 buffer is 60 seconds so A cannot meet a 500ms budget, and B recreates that batching by hand into a small-object mess. **@joudawad** · 1 upvotes > D — Sending 5,000 requests/second directly from 50,000 IoT devices to Lambda means 5,000 concurrent invocations per second with no buffer, no replay on failure, and no retention window if the anomaly detector falls behind. A Kinesis stream absorbs spikes and retains records — it is the missing buffer between device and compute. **@tomerl101** · 1 upvotes > Great post! > > C is the answer, but let's assume the company’s goal is to reduce costs at the expense of ingestion rate. In that case, would you go with Firehose, Data Streams, or neither? **@joudawad** · 1 upvotes > A — Firehose has no Lambda *(serverless compute)* destination. It can invoke a Lambda as a transformation step, but a transform runs inside the pipeline en route to a storage destination — it is not your consumer receiving each record. Even with zero buffering, delivery targets S3, Redshift, OpenSearch, or HTTP endpoints; the anomaly detector never gets the record on this path. --- Tags: [#aws](https://daily.dev/tags/aws), [#career](https://daily.dev/tags/career), [#data-engineering](https://daily.dev/tags/data-engineering) [View this post on daily.dev](https://daily.dev/posts/day-27-30-aws-system-design-patterns-zshlhvrvq) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/day-27-30-aws-system-design-patterns-zshlhvrvq","headline":"Day 27/30 AWS System Design Patterns","text":"A system design scenario walks through architecting an IoT data pipeline that must support both sub-500ms anomaly detection and cold storage archival to S3 as Parquet files. It compares Kinesis Data Streams and Kinesis Data Firehose, explaining why Firehose has no Lambda destination, why writing individual S3 objects from Streams creates small-file problems, and why direct Lambda invocation from devices lacks buffering. The correct design uses Kinesis Data Streams as the ingestion layer with two consumers: the anomaly detector reading directly from the stream, and a Firehose delivery stream subscribed as a second consumer to handle batching and S3 archival.","url":"https://daily.dev/posts/day-27-30-aws-system-design-patterns-zshlhvrvq","datePublished":"2026-08-24T20:47:30.737Z","dateModified":"2026-08-24T20:51:41.563Z","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0","description":"Principal Solution Architecture ","worksFor":{"@type":"Organization","name":"Metalab","logo":"https://www.google.com/s2/favicons?domain=metalab.com&sz=128"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":80110}},"image":"https://media.daily.dev/image/upload/s--hY4whoCP--/f_auto/v1787604700/posts/ZshlHvRvq?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":104},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":25}],"comment":[{"@type":"Comment","text":"The answer is C.\nKinesis Data Streams (real-time stream, sub-second consumer latency) and Kinesis Data Firehose (managed delivery pipe, buffers to S3/Redshift/OpenSearch) solve different problems and are designed to work together.\nKinesis Data Streams is a real-time streaming service. Consumers read from it with sub-second latency using shard iterators or enhanced fan-out. It retains records for 24 hours to 7 days and supports multiple independent consumers, each maintaining their own read position. This is the right tool for the anomaly detection service that needs records within 500 ms.\nKinesis Data Firehose is a fully managed delivery pipe. It buffers incoming records and delivers them to destinations — S3, Redshift, OpenSearch — in batches. It handles format conversion to Parquet, compression, and S3 partitioning automatically. It is not a consumer API. Buffering is configurable from 0 to 900 seconds — even zero-buffering delivery lands within a few seconds, but it lands in a destination like S3, not in your application code. This is the right tool for the S3 archive with 5-minute latency tolerance.\nFirehose can subscribe directly to a Kinesis Data Stream as its source. You create a Data Stream, the anomaly detector consumes from it in real time, and a Firehose delivery stream reads from the same Data Stream and batches to S3. One ingestion layer, two consumer paths, each optimized for its requirement.","datePublished":"2026-08-24T20:49:12.409Z","url":"https://daily.dev/posts/ZshlHvRvq#c-nABJpCazs","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":10}},{"@type":"Comment","text":"C, because Firehose’s minimum S3 buffer is 60 seconds so A cannot meet a 500ms budget, and B recreates that batching by hand into a small-object mess.","datePublished":"2026-08-31T20:25:32.738Z","url":"https://daily.dev/posts/ZshlHvRvq#c-ScGbKLbqk","author":{"@type":"Person","name":"Paolo Perrone","url":"https://daily.dev/paoloap","image":"https://lh3.googleusercontent.com/a/AGNmyxaOowFqs19Msmo4AtbOanvSwkOb705dhJ-u3fFQ0Nw=s96-c"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"D — Sending 5,000 requests/second directly from 50,000 IoT devices to Lambda means 5,000 concurrent invocations per second with no buffer, no replay on failure, and no retention window if the anomaly detector falls behind. A Kinesis stream absorbs spikes and retains records — it is the missing buffer between device and compute.","datePublished":"2026-08-24T20:48:21.090Z","url":"https://daily.dev/posts/ZshlHvRvq#c-hbJGsa3cA","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"Great post!\nC is the answer, but let’s assume the company’s goal is to reduce costs at the expense of ingestion rate. In that case, would you go with Firehose, Data Streams, or neither?","datePublished":"2026-08-25T17:14:20.558Z","dateModified":"2026-08-25T17:17:41.726Z","url":"https://daily.dev/posts/ZshlHvRvq#c-gV0I1PJwY","author":{"@type":"Person","name":"Tomer","url":"https://daily.dev/tomerl101","image":"https://avatars.githubusercontent.com/u/26263765?v=4"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"A — Firehose has no Lambda (serverless compute) destination. It can invoke a Lambda as a transformation step, but a transform runs inside the pipeline en route to a storage destination — it is not your consumer receiving each record. Even with zero buffering, delivery targets S3, Redshift, OpenSearch, or HTTP endpoints; the anomaly detector never gets the record on this path.","datePublished":"2026-08-24T20:48:09.627Z","url":"https://daily.dev/posts/ZshlHvRvq#c-9KOY2ecc5","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh","name":"Joud Awad"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Joud Awad","item":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh"},{"@type":"ListItem","position":3,"name":"Day 27/30 AWS System Design Patterns"}]} ```