--- title: Day 16/30 AWS System Design Patterns | daily.dev description: A real-time sports scoreboard built on AWS AppSync (GraphQL WebSocket subscriptions) and DynamoDB experiences a thundering herd problem during a championship... canonical: https://daily.dev/posts/day-16-30-aws-system-design-patterns-qs6ncb15o twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Day 16/30 AWS System Design Patterns | daily.dev og:description: A real-time sports scoreboard built on AWS AppSync (GraphQL WebSocket subscriptions) and DynamoDB experiences a thundering herd problem during a championship... og:url: https://daily.dev/posts/day-16-30-aws-system-design-patterns-qs6ncb15o og:image: https://api.daily.dev/og/posts/qS6NCB15o.png og:image:alt: Day 16/30 AWS System Design Patterns og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Day 16/30 AWS System Design Patterns **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 2 min read · 102 upvotes · 25 comments ## Summary A real-time sports scoreboard built on AWS AppSync (GraphQL WebSocket subscriptions) and DynamoDB experiences a thundering herd problem during a championship final. When 95,000 users open the app simultaneously within 90 seconds, connection attempts peak at ~3,500/s — exceeding AppSync's default 2,000 connection requests/second quota. Rejected clients retry immediately, sustaining the overload and leaving 30% of viewers with frozen scoreboards. The correct fix combines a quota increase, client-side jittered exponential backoff, and connection sharding across multiple AppSync APIs. ## Content A live sports platform uses AppSync _(managed GraphQL API service — real-time subscriptions over WebSocket, with service quotas on connection and message rates)_ to power a real-time scoreboard. Fans open the app during a match and subscribe to `onScoreUpdate`, backed by a DynamoDB _(managed NoSQL database)_ table. Tonight is the championship final. At kickoff, the platform sends a push notification to 400,000 users: "The final has started." Within 90 seconds, 95,000 people open the app, and every one of those clients immediately attempts a WebSocket connection — a synchronized surge peaking around 3,500 connection attempts per second. Eight minutes in, roughly 30% of viewers have a frozen scoreboard. The other 70% receive every update normally. Application logs show no errors. AppSync metrics show a spike in `4xxError` responses concentrated in the connection surge, and it isn't decaying — the affected clients are retrying their connections immediately on failure, in a tight loop. What is the most likely cause and the correct architectural response? A) AppSync's WebSocket idle timeout is dropping connections and the clients aren't sending keepalives — but AppSync itself sends periodic keep-alive messages to maintain established connections, and an idle-drop failure would hit long-connected clients over time, not concentrate 4xx errors in the first minutes of a connection surge B) An ALB _(Application Load Balancer — customer-managed, and not part of AppSync's path)_ in front of AppSync is rejecting the surge — but AppSync manages its own WebSocket infrastructure end to end; there is no customer-controlled load balancer in front of it to tune C) The surge exceeded AppSync's connection request rate quota _(a per-API service quota on new WebSocket connection requests per second — default 2,000/s, adjustable)_ — attempts over the rate are rejected with 4xx, and the clients' instant retries sustained a thundering herd that kept the rate pegged over quota; the fix is a quota increase, client-side jittered backoff, and connection sharding across APIs D) DynamoDB Streams _(change feed from DynamoDB, drives the subscription publish)_ cannot fan out to 95,000 subscribers — but one score update triggers one publish that AppSync fans out to all connected clients, and a fan-out bottleneck would degrade delivery for everyone, not cleanly split the audience into connected and never-connected Answer in the comments. #AWS #SystemDesign #RealTime #Serverless #DynamoDB ## Community discussion Top comments from developers on daily.dev. **@joudawad** · 4 upvotes > The answer is C. > > AppSync _(managed GraphQL API — quotas govern its real-time layer)_ protects its real-time infrastructure with rate quotas, and the one that matters at kickoff is the **connection request rate**: a per-API ceiling on new WebSocket connection attempts per second (default 2,000/s, adjustable through Service Quotas). It is a rate, not a headcount — AppSync happily holds hundreds of thousands of concurrent connections. What it will not do is _accept_ them faster than the quota. > > A push notification is a synchronization device. Sending "the final has started" to 400,000 phones... **@yk90** · 2 upvotes > Not an AWS Expert, but two questions come to mind for this case. > > The first major innovations you learn from the AWS beginner certificates is the auto-scalability, elasticity, the concept of bursts and traffic spikes as core concepts the platform started from. I know those mainly started off the EC2 Instances concept and may architecturally and design-wise not be feasible to replicate to other services, but 1. Does having similar concepts available and applicable for other services such as AppSync translate to a QoL feature similar clients and users would appreciate given they actually would... **@ragu** · 1 upvotes > Use full post!! **@apperside** · 1 upvotes > These articles series is really awesome **@agustinbarrientos** · 1 upvotes > That 2,000-per-second quota turns the 400,000-user kickoff into a nasty load test ## Similar posts on daily.dev - [The ‘Super Bowl’ standard: Architecting distributed systems for massive concurrency](https://daily.dev/posts/the-super-bowl-standard-architecting-distributed-systems-for-massive-concurrency-mvxs8yfr2) · InfoWorld · 0 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#career](https://daily.dev/tags/career), [#aws-dynamodb](https://daily.dev/tags/aws-dynamodb), [#websocket](https://daily.dev/tags/websocket) [View this post on daily.dev](https://daily.dev/posts/day-16-30-aws-system-design-patterns-qs6ncb15o) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/day-16-30-aws-system-design-patterns-qs6ncb15o","headline":"Day 16/30 AWS System Design Patterns","text":"A real-time sports scoreboard built on AWS AppSync (GraphQL WebSocket subscriptions) and DynamoDB experiences a thundering herd problem during a championship final. When 95,000 users open the app simultaneously within 90 seconds, connection attempts peak at ~3,500/s — exceeding AppSync's default 2,000 connection requests/second quota. Rejected clients retry immediately, sustaining the overload and leaving 30% of viewers with frozen scoreboards. The correct fix combines a quota increase, client-side jittered exponential backoff, and connection sharding across multiple AppSync APIs.","url":"https://daily.dev/posts/day-16-30-aws-system-design-patterns-qs6ncb15o","datePublished":"2026-08-05T17:38:08.861Z","dateModified":"2026-09-13T21:22:36.896Z","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0","description":"Principal Solution Architecture ","worksFor":{"@type":"Organization","name":"Metalab","logo":"https://www.google.com/s2/favicons?domain=metalab.com&sz=128"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":86200}},"image":"https://media.daily.dev/image/upload/s--Wl19tvIQ--/f_auto/v1785951494/posts/qS6NCB15o?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":102},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":25}],"comment":[{"@type":"Comment","text":"The answer is C.\nAppSync (managed GraphQL API — quotas govern its real-time layer) protects its real-time infrastructure with rate quotas, and the one that matters at kickoff is the connection request rate: a per-API ceiling on new WebSocket connection attempts per second (default 2,000/s, adjustable through Service Quotas). It is a rate, not a headcount — AppSync happily holds hundreds of thousands of concurrent connections. What it will not do is accept them faster than the quota.\nA push notification is a synchronization device. Sending “the final has started” to 400,000 phones converts a normally smeared-out arrival pattern into a spike: 3,500 connection attempts per second against a 2,000/s budget. The first 2,000 each second get in — that’s the healthy 70%, who connected and stayed connected. The overflow is rejected with a 4xx.\nThen the second mechanism takes over: every rejected client retries immediately. Failed attempts re-enter the same rate budget, so the herd competes with itself, holding the attempt rate above quota long after the organic surge would have passed. This is the thundering herd, and it is self-sustaining — the 4xx spike “not decaying” in the metrics is its signature.\nThree fixes, in order of leverage:\nFix 1 — Client-side jittered exponential backoff on connection failures. This is the cheapest and most important change: it spreads the herd back out in time, so the same quota admits everyone within seconds instead of rejecting the same 30% forever. (AWS’s own real-time protocol guidance says exactly this.) While you’re in the client, stagger the initial connect too — a random 0–15 second delay after the push notification flattens the spike at its source.\nFix 2 — Request the quota increase before the next final, sized to your worst 1-second burst, not your average. Quota increases are a support workflow, not a dial — they take days, not minutes.\nFix 3 — Connection sharding: route user segments (consistent hash, region, device) to separate AppSync APIs, multiplying your admission rate across quota pools. This is also your scale path beyond any single quota.\nThe pattern outlives AppSync: any push notification, cache expiry, or deploy that synchronizes client behavior converts average load into peak load, and every admission-controlled system — WebSocket fleets, connection pools, rate-limited APIs — meets that peak with rejection. Correlated arrivals plus naive retries is how systems stay down after the original spike has passed. Jitter is not a nicety; it is the antidote.\nWorth evaluating at this scale: for broadcast-style delivery — one identical update to 95,000 viewers — AppSync Events (HTTP pub/sub API, lighter per-connection model than GraphQL subscriptions) is a better-fitted primitive than resolver-backed GraphQL subscriptions. Same quota discipline applies.","datePublished":"2026-08-05T17:39:07.388Z","url":"https://daily.dev/posts/qS6NCB15o#c-cuUkgJ5Wz","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":4}},{"@type":"Comment","text":"Not an AWS Expert, but two questions come to mind for this case.\nThe first major innovations you learn from the AWS beginner certificates is the auto-scalability, elasticity, the concept of bursts and traffic spikes as core concepts the platform started from. I know those mainly started off the EC2 Instances concept and may architecturally and design-wise not be feasible to replicate to other services, but 1. Does having similar concepts available and applicable for other services such as AppSync translate to a QoL feature similar clients and users would appreciate given they actually would be a solution for the root causes of such cases. 2. Are there any AI analytics available of AWS that can be leveraged for risk forecasting on such availability risk, real-time analytics for root-cause analysis and auto-triage and resolutions with HITL graphs? and actually a 3rd one, 3. Re-attempts from rejected users from same origins causing an ever growing pile of over-quota usage seems a bad practice? Is there a reason it cannot detect a re-attempt from a failed first attempt by default? (privacy, data collection or any other restrictions)","datePublished":"2026-08-05T18:11:42.916Z","dateModified":"2026-08-05T18:16:44.177Z","url":"https://daily.dev/posts/qS6NCB15o#c-Z10dB0SRT","author":{"@type":"Person","name":"Yehya Karout","url":"https://daily.dev/yk90","image":"https://lh3.googleusercontent.com/a/ACg8ocKuxiOD2pMMkjDJDuYEiza2CljWaODJUi4ObI4XjozQNy32SA=s96-c"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"Use full post!!","datePublished":"2026-08-07T09:03:04.963Z","url":"https://daily.dev/posts/qS6NCB15o#c-3MlzVjfuP","author":{"@type":"Person","name":"Ragunathan Maniraj","url":"https://daily.dev/ragu","image":"https://avatars.githubusercontent.com/u/13572570?v=4"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"These articles series is really awesome","datePublished":"2026-08-07T02:04:13.074Z","url":"https://daily.dev/posts/qS6NCB15o#c-hXbmZwjid","author":{"@type":"Person","name":"Apperside","url":"https://daily.dev/apperside","image":"https://avatars.githubusercontent.com/u/5955338?v=4"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"That 2,000-per-second quota turns the 400,000-user kickoff into a nasty load test","datePublished":"2026-08-06T21:54:13.159Z","url":"https://daily.dev/posts/qS6NCB15o#c-BwqYk6hL7","author":{"@type":"Person","name":"Agustin Barrientos","url":"https://daily.dev/agustinbarrientos","image":"https://media.daily.dev/image/upload/s--5ayxQnqn--/f_auto/v1788281802/avatars/avatar_wQYYVe5Tbj0NJ7C7qPoa8?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh","name":"Joud Awad"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Joud Awad","item":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh"},{"@type":"ListItem","position":3,"name":"Day 16/30 AWS System Design Patterns"}]} ```