A Kinesis Data Stream (real-time stream, 2 MB/s + 5 transactions/sec read limit per shard — shared across all consumers using standard GetRecords) ingests clickstream events for a real-time analytics platform. The stream has 8 shards. Three consumer applications read from it: an anomaly detector, a session aggregator, and a dashboard feed. All three use the standard <code>GetRecords</code> API via polling.
At peak, the stream receives 1.2 million events per hour across all shards. Each shard produces roughly 150,000 records per hour — about 1.5 MB/s of data, within the 2 MB/s read limit per shard.
The dashboard feed consumer starts falling behind. Its <code>IteratorAgeMilliseconds</code> metric climbs to 45 seconds during peak hours. The anomaly detector and session aggregator are keeping up fine.
The team checks: no single shard is over the throughput limit. Total read consumption is 1.8 MB/s per shard — across all three consumers combined. Per-shard read limit is 2 MB/s. They are under the limit.
Why is the dashboard consumer falling behind?
A) The dashboard consumer’s Lambda function (serverless compute) is undersized — it cannot process records fast enough and is falling behind at the application level; but <code>IteratorAgeMilliseconds</code> climbing during peak while the other two consumers are fine points to a read throughput problem, not a processing speed problem
B) All three consumers share the 2 MB/s per-shard read limit — with three consumers polling the same shard via standard GetRecords, each gets roughly 667 KB/s; a shard producing 1.5 MB/s but delivering only 667 KB/s to one consumer means that consumer falls behind
C) The dashboard consumer needs its own dedicated shard (shards are the unit of write partitioning — all consumers read from all shards; you cannot assign a shard to a single consumer) — shared shards cause head-of-line blocking between consumers
D) The dashboard consumer should use Kinesis Firehose (managed delivery pipe, buffers records 60s–900s before delivering to S3/Redshift — not a real-time consumer API) instead of reading directly from the Data Stream
Answer in the comments.
#EventDriven #SystemDesgin #AWS #Serverless #Architecture

Smiley Face

Joud Awad

A system design scenario involving Amazon Kinesis Data Streams explores why a dashboard consumer falls behind during peak load despite all shards being under the 2 MB/s throughput limit. With 8 shards and three consumers using the standard GetRecords API, the key insight is that the 2 MB/s per-shard read limit is shared across all consumers polling that shard — meaning each of the three consumers effectively gets only ~667 KB/s per shard, while each shard produces 1.5 MB/s. This causes the dashboard consumer to fall behind, even though no individual shard appears overloaded.

Day 19/30 AWS System Design Patterns

yep, that shared 2MB/s cap per shard is exactly why AWS added enhanced fan-out, each consumer gets its own dedicated 2MB/s pipe over HTTP/2 push instead of splitting the shared GetRecords budget. costs more per consumer-shard but it’s the actual fix instead of over-provisioning shards.

The answer is B.
Kinesis Data Streams (real-time stream) enforces a 2 MB/s and 5 transactions/second read limit per shard — and this limit is shared across all consumers reading from that shard using the standard <code>GetRecords</code> API. With three consumers polling the same shard, each consumer’s effective read throughput is throttled to roughly one-third of the shard’s capacity. Each individual consumer is getting around 600 KB/s from a shard producing 1.5 MB/s. That consumer will fall behind.
The correct fix is Kinesis enhanced fan-out (dedicated 2 MB/s read pipe per registered consumer — independent of other consumers). When you register a consumer with enhanced fan-out, that consumer gets a dedicated 2 MB/s throughput pipe per shard, independent of every other consumer. The three consumers no longer compete for the same read budget. Enhanced fan-out uses a push model (HTTP/2 streaming) instead of polling — consumers also see lower latency: typically under 70 ms from record ingestion to delivery, compared to 200–400 ms with standard polling.
<img src="https://media.daily.dev/image/upload/s--B0tDFM17--/f_auto/v1786211982/ugc/content_05c11c65-525c-4306-a2ca-88cbbf507551?_a=BAMAMicg0" alt="ChatGPT Image Aug 8, 2026, 08_59_31 PM.png">

B is the key here: the limit is shared per shard, not granted separately to each polling consumer. Enhanced fan-out would give the dashboard its own 2 MB/s read pipe, but I’d still watch IteratorAge alongside processing duration so a slow Lambda doesn’t hide behind the same symptom.

Superb <a href="https://daily.dev/joudawad" data-mention-id="iaC4JsBU0lV8wBsc85fSh" data-mention-username="joudawad" translate="no">@joudawad</a> ! I salute to your consistency!