--- title: Day 6/30 AWS System Design Patterns | daily.dev description: A daily AWS system design challenge exploring a rolling deployment scenario where EC2 Auto Scaling and ALB deregistration delay are misconfigured, causing 502... canonical: https://daily.dev/posts/day-6-30-aws-system-design-patterns-c2sy3vziq twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Day 6/30 AWS System Design Patterns | daily.dev og:description: A daily AWS system design challenge exploring a rolling deployment scenario where EC2 Auto Scaling and ALB deregistration delay are misconfigured, causing 502... og:url: https://daily.dev/posts/day-6-30-aws-system-design-patterns-c2sy3vziq og:image: https://api.daily.dev/og/posts/c2sY3vziQ.png og:image:alt: Day 6/30 AWS System Design Patterns og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Day 6/30 AWS System Design Patterns **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 2 min read · 82 upvotes · 17 comments ## Summary A daily AWS system design challenge exploring a rolling deployment scenario where EC2 Auto Scaling and ALB deregistration delay are misconfigured, causing 502 errors during instance termination. The puzzle asks readers to identify the root cause: missing EC2 Auto Scaling lifecycle hooks that allow instances to be terminated before the ALB draining window completes, versus other plausible but incorrect explanations like insufficient deregistration delay, SIGTERM mishandling, or an ALB routing bug. ## Content A SaaS platform uses EC2 Auto Scaling _(automatically launches or terminates EC2 instances based on demand)_ to run its API tier. Instances run the application server, hold in-memory session state for active requests, and drain gracefully on shutdown by finishing in-flight requests before terminating. The platform deploys a new version via a rolling update. The Auto Scaling group terminates old instances and launches new ones. During each termination event, the ALB _(Application Load Balancer, stops routing new requests to draining targets)_ deregisters the instance and marks it as draining — but active requests that were mid-flight at the moment of deregistration return 502 errors to clients. The engineering team increases the ALB deregistration delay from 30 seconds to 120 seconds. The 502 errors continue. What is the actual problem? A) The ALB deregistration delay _(tells ALB to wait N seconds before fully deregistering — allows in-flight requests to complete)_ is still too short — increase it to 300 seconds to give all in-flight requests time to complete; but the 502s continued after increasing from 30 to 120 seconds — if delay length were the only problem, the errors would have decreased, not continued unchanged B) The EC2 Auto Scaling lifecycle hook _(pauses instance termination until your code signals completion — coordinates ALB drain timing with actual EC2 termination)_ is not configured — without it, instances are terminated immediately after receiving the termination signal, before the ALB deregistration delay can drain connections C) The application server is not handling SIGTERM _(Linux signal sent to process on termination — process should finish in-flight work then exit cleanly)_ — it exits immediately on the termination signal instead of waiting for in-flight requests to finish D) The ALB _(Application Load Balancer)_ is sending new requests to the draining instance during the deregistration window — this is a known ALB bug at high connection counts; but ALBs do not route new requests to targets in draining state — this is not a bug Answer in the comments. **#AWS #SystemDesign #DevOps #CloudArchitecture #SRE** ## Community discussion Top comments from developers on daily.dev. **@joudawad** · 20 upvotes > The answer is B. > > The ALB _(Application Load Balancer)_ deregistration delay and the EC2 instance termination lifecycle are two separate, independent timers — and they are not coordinated by default. > > When Auto Scaling _(automatically launches or terminates EC2 instances)_ decides to terminate an instance, it sends a termination signal. Without a lifecycle hook, the instance is terminated almost immediately — in practice within seconds. The ALB deregistration delay (set to 120 seconds) tells the ALB to stop sending new requests and wait for in-flight requests to drain. But if the EC2 instance... **@joudawad** · 7 upvotes > C — SIGTERM *(Linux process termination signal)* handling is worth verifying independently. But if SIGTERM were the only problem, increasing the deregistration delay would have reduced errors. The fact that 502s continue unchanged after the delay increase points to the instance being gone before the drain window even starts. **@joudawad** · 7 upvotes > A — Increasing the deregistration delay to 300 seconds does not fix the problem because the instance is being terminated before the delay can do anything. The root cause is the missing lifecycle hook that coordinates timing between ALB drain and EC2 termination. **@joudawad** · 6 upvotes > Also, it would mean a lot to me if you could support my content and stay in touch 🙏 > > - YouTube: [https://www.youtube.com/@system-design-lab](https://www.youtube.com/@system-design-lab) > - LinkedIn: [https://www.linkedin.com/in/joud-awad/](https://www.linkedin.com/in/joud-awad/) > - Medium Blog: [https://joudwawad.medium.com/](https://joudwawad.medium.com/) > - Substack: [https://joudawad.substack.com/](https://joudawad.substack.com/) **@joudawad** · 5 upvotes > D — ALBs do not route new requests to targets in draining state. Deregistration correctly stops new request routing while allowing in-flight requests to complete — assuming the instance is still running. --- Tags: [#aws](https://daily.dev/tags/aws), [#career](https://daily.dev/tags/career) [View this post on daily.dev](https://daily.dev/posts/day-6-30-aws-system-design-patterns-c2sy3vziq) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/day-6-30-aws-system-design-patterns-c2sy3vziq","headline":"Day 6/30 AWS System Design Patterns","text":"A daily AWS system design challenge exploring a rolling deployment scenario where EC2 Auto Scaling and ALB deregistration delay are misconfigured, causing 502 errors during instance termination. The puzzle asks readers to identify the root cause: missing EC2 Auto Scaling lifecycle hooks that allow instances to be terminated before the ALB draining window completes, versus other plausible but incorrect explanations like insufficient deregistration delay, SIGTERM mishandling, or an ALB routing bug.","url":"https://daily.dev/posts/day-6-30-aws-system-design-patterns-c2sy3vziq","datePublished":"2026-07-26T12:15:10.182Z","dateModified":"2026-09-13T20:57:23.594Z","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0","description":"Principal Solution Architecture ","worksFor":{"@type":"Organization","name":"Metalab","logo":"https://www.google.com/s2/favicons?domain=metalab.com&sz=128"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":86150}},"image":"https://media.daily.dev/image/upload/s--qJJqaTEu--/f_auto/v1785068111/posts/c2sY3vziQ?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":82},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":17}],"comment":[{"@type":"Comment","text":"The answer is B.\nThe ALB (Application Load Balancer) deregistration delay and the EC2 instance termination lifecycle are two separate, independent timers — and they are not coordinated by default.\nWhen Auto Scaling (automatically launches or terminates EC2 instances) decides to terminate an instance, it sends a termination signal. Without a lifecycle hook, the instance is terminated almost immediately — in practice within seconds. The ALB deregistration delay (set to 120 seconds) tells the ALB to stop sending new requests and wait for in-flight requests to drain. But if the EC2 instance is already terminated before those 120 seconds elapse, there is nothing left to drain. The ALB’s drain window is irrelevant because the instance is gone.\nEC2 Auto Scaling lifecycle hooks (pause instance termination until your code signals completion) solve this. A autoscaling:EC2INSTANCETERMINATING lifecycle hook puts the instance into a Terminating:Wait state, pausing the actual termination. Your drain logic runs — waiting for in-flight requests to complete and for the ALB deregistration delay to elapse — then sends CompleteLifecycleAction to allow termination to proceed.\nThe correct configuration: lifecycle hook on termination → hook handler waits for drain to complete → sends CompleteLifecycleAction → Auto Scaling terminates the instance.","datePublished":"2026-07-26T12:15:34.171Z","url":"https://daily.dev/posts/c2sY3vziQ#c-qy49BI695","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":20}},{"@type":"Comment","text":"C — SIGTERM (Linux process termination signal) handling is worth verifying independently. But if SIGTERM were the only problem, increasing the deregistration delay would have reduced errors. The fact that 502s continue unchanged after the delay increase points to the instance being gone before the drain window even starts.","datePublished":"2026-07-26T12:15:50.839Z","url":"https://daily.dev/posts/c2sY3vziQ#c-q28Qp4yH4","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":7}},{"@type":"Comment","text":"A — Increasing the deregistration delay to 300 seconds does not fix the problem because the instance is being terminated before the delay can do anything. The root cause is the missing lifecycle hook that coordinates timing between ALB drain and EC2 termination.","datePublished":"2026-07-26T12:15:45.068Z","url":"https://daily.dev/posts/c2sY3vziQ#c-QNsfcLEgE","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":7}},{"@type":"Comment","text":"Also, it would mean a lot to me if you could support my content and stay in touch 🙏\n\nYouTube: https://www.youtube.com/@system-design-lab\nLinkedIn: https://www.linkedin.com/in/joud-awad/\nMedium Blog: https://joudwawad.medium.com/\nSubstack: https://joudawad.substack.com/","datePublished":"2026-07-26T12:16:05.277Z","url":"https://daily.dev/posts/c2sY3vziQ#c-TRC23iXCC","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":6}},{"@type":"Comment","text":"D — ALBs do not route new requests to targets in draining state. Deregistration correctly stops new request routing while allowing in-flight requests to complete — assuming the instance is still running.","datePublished":"2026-07-26T12:15:56.300Z","url":"https://daily.dev/posts/c2sY3vziQ#c-OaLPIi7jv","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":5}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh","name":"Joud Awad"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Joud Awad","item":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh"},{"@type":"ListItem","position":3,"name":"Day 6/30 AWS System Design Patterns"}]} ```