--- title: Day 1/30 AWS System Design Patterns | daily.dev description: A gaming platform leaderboard stored in DynamoDB experiences ThrottlingException errors and high P99 latency on reads for a single dominant game... canonical: https://daily.dev/posts/day-1-30-aws-system-design-patterns-mkhcecn0y twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Day 1/30 AWS System Design Patterns | daily.dev og:description: A gaming platform leaderboard stored in DynamoDB experiences ThrottlingException errors and high P99 latency on reads for a single dominant game... og:url: https://daily.dev/posts/day-1-30-aws-system-design-patterns-mkhcecn0y og:image: https://api.daily.dev/og/posts/mKHCeCn0Y.png og:image:alt: Day 1/30 AWS System Design Patterns og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Day 1/30 AWS System Design Patterns **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 2 min read · 293 upvotes · 67 comments ## Summary A gaming platform leaderboard stored in DynamoDB experiences ThrottlingException errors and high P99 latency on reads for a single dominant game ('battle-royale'), despite CloudWatch showing consumed RCUs well below provisioned capacity. Doubling provisioned RCUs fails to resolve the issue. The root cause is a hot partition: when traffic concentrates on a single partition key, that partition hits its own throughput ceiling regardless of total table-level RCU headroom. The correct answer is C — per-partition limits apply independently of aggregate table provisioning. ## Content A gaming platform stores player leaderboard data in DynamoDB _(managed NoSQL database, distributes data across internal partitions — each partition has its own throughput ceiling)_. The table uses `gameid`_ as the partition key and _`playerid` as the sort key. The platform runs 15 active games. One game — `game_id = "battle-royale"` — accounts for 78% of all read traffic. It is the flagship title that launched 6 months ago and still dominates the player base. The table is provisioned at 10,000 RCUs (Read Capacity Units distributed across all the partitions, ) . CloudWatch _(AWS monitoring service, reports consumed RCUs as a table-level aggregate across all partitions)_ shows average consumed RCUs at 4,300 — well under the provisioned capacity. But `ThrottlingException` errors are spiking on leaderboard reads for `battle-royale` during peak hours, and P99 read latency has climbed to 820 ms. The on-call engineer opens a ticket to increase provisioned capacity to 20,000 RCUs. The support team approves the change. It goes live. The throttling does not stop. CloudWatch still shows consumed RCUs well below the new limit. You have 10,000 RCUs of headroom sitting idle and a table that is still throttling. What is actually happening? A) Yes — doubling provisioned RCUs gives the table enough capacity headroom to absorb the battle-royale traffic spike B) No — the table needs to migrate to DynamoDB on-demand capacity mode _(no pre-provisioned capacity, adapts per-partition throughput dynamically)_, which removes per-partition limits entirely C) No — battle-royale is a hot partition; per-partition throughput limits apply regardless of total table RCU provisioning; traffic concentrated on one partition key hits that partition's ceiling even when the table has headroom D) No — the issue is a missing GSI _(Global Secondary Index — a secondary index for alternate access patterns)_ on `game_id` causing full partition scans on leaderboard reads #SystemDesign #AWS #DynamoDB #SoftwareEngineering #Backend ## Community discussion Top comments from developers on daily.dev. **@joudawad** · 17 upvotes > The answer is C. > > DynamoDB _(managed NoSQL database)_ does not store all your data in one place. It distributes data across internal partitions based on the partition key hash. Each partition has its own throughput ceiling: 3,000 RCUs and 1,000 WCUs. These limits are per-partition — not per-table. > > When `battle-royale` receives 78% of all read traffic, that traffic concentrates on the partition(s) storing `game_id = "battle-royale"` data. When that partition's 3,000 RCU ceiling is hit, DynamoDB throttles reads on it. The other partitions — the ones holding the other 14 games — still have... **@joudawad** · 13 upvotes > Also, it would mean a lot to me if you could support my content and stay in touch 🙏 > > - YouTube: [https://www.youtube.com/@system-design-lab](https://www.youtube.com/@system-design-lab) > - LinkedIn: [https://www.linkedin.com/in/joud-awad/](https://www.linkedin.com/in/joud-awad/) > - Medium Blog: [https://joudwawad.medium.com/](https://joudwawad.medium.com/) > - Substack: [https://joudawad.substack.com/](https://joudawad.substack.com/) **@joudawad** · 4 upvotes > A — The logic feels sound: more provisioned RCUs means more headroom. But CloudWatch's _(AWS monitoring service)_ consumed RCU metric is a table-level aggregate — it sums consumption across all partitions. It shows 4,300 consumed out of 10,000 because the 14 other games are barely loaded. The single partition holding `battle-royale` data has already hit its 3,000 RCU per-partition ceiling. Doubling total RCUs to 20,000 does not redistribute that partition's load. The same requests still hit the same partition. **@markozakrajsekgeozs** · 4 upvotes > I come to the [daily.dev](https://daily.dev), just because someone pointed out your articles to me! Thanks for all the knowledge. **@jad85** · 3 upvotes > WooooHoooo!!! Here we go again!!! --- Tags: [#aws](https://daily.dev/tags/aws), [#career](https://daily.dev/tags/career), [#backend](https://daily.dev/tags/backend), [#aws-dynamodb](https://daily.dev/tags/aws-dynamodb) [View this post on daily.dev](https://daily.dev/posts/day-1-30-aws-system-design-patterns-mkhcecn0y) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/day-1-30-aws-system-design-patterns-mkhcecn0y","headline":"Day 1/30 AWS System Design Patterns","text":"A gaming platform leaderboard stored in DynamoDB experiences ThrottlingException errors and high P99 latency on reads for a single dominant game ('battle-royale'), despite CloudWatch showing consumed RCUs well below provisioned capacity. Doubling provisioned RCUs fails to resolve the issue. The root cause is a hot partition: when traffic concentrates on a single partition key, that partition hits its own throughput ceiling regardless of total table-level RCU headroom. The correct answer is C — per-partition limits apply independently of aggregate table provisioning.","url":"https://daily.dev/posts/day-1-30-aws-system-design-patterns-mkhcecn0y","datePublished":"2026-07-21T15:28:25.625Z","dateModified":"2026-07-22T04:17:31.624Z","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0","description":"Principal Solution Architecture ","worksFor":{"@type":"Organization","name":"Metalab","logo":"https://www.google.com/s2/favicons?domain=metalab.com&sz=128"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":81900}},"image":"https://media.daily.dev/image/upload/s--veE8yXvE--/f_auto/v1784647711/posts/mKHCeCn0Y?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":293},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":67}],"comment":[{"@type":"Comment","text":"The answer is C.\nDynamoDB (managed NoSQL database) does not store all your data in one place. It distributes data across internal partitions based on the partition key hash. Each partition has its own throughput ceiling: 3,000 RCUs and 1,000 WCUs. These limits are per-partition — not per-table.\nWhen battle-royale receives 78% of all read traffic, that traffic concentrates on the partition(s) storing game_id = "battle-royale" data. When that partition’s 3,000 RCU ceiling is hit, DynamoDB throttles reads on it. The other partitions — the ones holding the other 14 games — still have capacity available. CloudWatch’s (AWS monitoring service) consumed RCU metric is a table-level aggregate across all partitions. It shows 4,300 consumed out of 10,000 because the 14 other games are barely loaded. The hot partition is buried in that average.\nDoubling total provisioned RCUs to 20,000 does not move the ceiling on the hot partition. The same battle-royale requests still hit the same partition. The per-partition ceiling still applies.\nTwo correct fixes:\nOption 1 — Migrate to DynamoDB on-demand mode (no pre-provisioned capacity, adapts per-partition throughput dynamically). On-demand adapts throughput per partition dynamically and handles burst traffic better than provisioned mode. It does not remove per-partition limits entirely, but it responds to sudden spikes faster than provisioned auto-scaling.\nOption 2 — Add a random suffix to the partition key: battle-royale#0 through battle-royale#9. Scatter leaderboard writes across 10 logical partitions. Use a scatter-gather read (parallel queries on all 10 suffixes, merge and rank in application code) to reconstruct the leaderboard. More complex operationally, but it eliminates the hot partition by design.","datePublished":"2026-07-21T15:29:27.402Z","url":"https://daily.dev/posts/mKHCeCn0Y#c-wDiDp4Zdw","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":17}},{"@type":"Comment","text":"Also, it would mean a lot to me if you could support my content and stay in touch 🙏\n\nYouTube: https://www.youtube.com/@system-design-lab\nLinkedIn: https://www.linkedin.com/in/joud-awad/\nMedium Blog: https://joudwawad.medium.com/\nSubstack: https://joudawad.substack.com/","datePublished":"2026-07-21T15:28:58.002Z","url":"https://daily.dev/posts/mKHCeCn0Y#c-BoZtIESiN","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":13}},{"@type":"Comment","text":"A — The logic feels sound: more provisioned RCUs means more headroom. But CloudWatch’s (AWS monitoring service) consumed RCU metric is a table-level aggregate — it sums consumption across all partitions. It shows 4,300 consumed out of 10,000 because the 14 other games are barely loaded. The single partition holding battle-royale data has already hit its 3,000 RCU per-partition ceiling. Doubling total RCUs to 20,000 does not redistribute that partition’s load. The same requests still hit the same partition.","datePublished":"2026-07-21T15:29:30.334Z","url":"https://daily.dev/posts/mKHCeCn0Y#c-UCz7Cnmyz","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":4}},{"@type":"Comment","text":"I come to the daily.dev, just because someone pointed out your articles to me! Thanks for all the knowledge.","datePublished":"2026-07-22T05:23:33.156Z","url":"https://daily.dev/posts/mKHCeCn0Y#c-gBuNXAKR6","author":{"@type":"Person","name":"Marko Zakrajsek","url":"https://daily.dev/markozakrajsekgeozs","image":"https://media.daily.dev/image/upload/s--slY3-STy--/f_auto/v1783585373/avatars/avatar_SbncGJHaRHf7KoebEkT1L?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":4}},{"@type":"Comment","text":"WooooHoooo!!! Here we go again!!!","datePublished":"2026-07-21T15:37:20.963Z","url":"https://daily.dev/posts/mKHCeCn0Y#c-cjjJMYLv8","author":{"@type":"Person","name":"Jad","url":"https://daily.dev/jad85"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh","name":"Joud Awad"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Joud Awad","item":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh"},{"@type":"ListItem","position":3,"name":"Day 1/30 AWS System Design Patterns"}]} ```