--- title: Day 9/30 AWS System Design Patterns | daily.dev description: A scenario-based AWS system design puzzle involving a Lambda durable function powering a loan approval workflow. The workflow downloads a 40 MB compliance... canonical: https://daily.dev/posts/day-9-30-aws-system-design-patterns-ovlngbuvs twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Day 9/30 AWS System Design Patterns | daily.dev og:description: A scenario-based AWS system design puzzle involving a Lambda durable function powering a loan approval workflow. The workflow downloads a 40 MB compliance... og:url: https://daily.dev/posts/day-9-30-aws-system-design-patterns-ovlngbuvs og:image: https://api.daily.dev/og/posts/oVLngbUvs.png og:image:alt: Day 9/30 AWS System Design Patterns og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Day 9/30 AWS System Design Patterns **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 2 min read · 83 upvotes · 15 comments ## Summary A scenario-based AWS system design puzzle involving a Lambda durable function powering a loan approval workflow. The workflow downloads a 40 MB compliance ruleset from S3 outside any checkpointed step, causing it to re-execute on every replay. As the checkpoint log grows with each review cycle, the 50-second S3 load plus replay overhead eventually exceeds the 90-second Lambda function timeout, freezing 1,900 applications mid-review. The correct diagnosis (option C) highlights a classic durable execution pitfall: non-idempotent or expensive code placed outside durable steps is not checkpointed and re-runs on every resume, compounding cost with each replay. ## Content A lending platform runs its loan approval workflow as a Lambda durable function _(Lambda's checkpoint-and-replay execution mode — on every resume, the handler re-runs from the top and skips completed durable operations using their stored results)_. The flow: validate the application (a `context.step()`), pull the credit report (a step), then a review loop — `context.wait()` for 6 hours, check whether a human reviewer has approved, repeat for up to 5 days. Function timeout: 90 seconds per invocation. Execution timeout: 7 days. At the top of the handler — before any step — the code downloads a 40 MB compliance ruleset from S3 _(object storage)_ and parses it. It takes about 50 seconds. The engineer put it there deliberately: "several steps need it, so load it once." The workflow ships. Applications flow through reviews for two days. Then, gradually, applications stop progressing. By Thursday, 1,900 applications are frozen mid-review. The business logic has thrown zero errors. The durable execution history shows executions alive and well within their 7-day window — just never advancing past their latest wait. The only anomaly: invocation duration on the function is pinned at exactly 90 seconds. What is happening? A) The executions exceeded the durable execution timeout — but the execution timeout (7 days here, up to 1 year) is a separate setting from the Lambda function timeout, and the history shows executions only 2–3 days old and still active B) `context.wait()` keeps the function running and billing during the 6-hour waits, and the accumulated wait time consumed the timeout — but a wait suspends the execution entirely; each resume is a brand-new invocation with a fresh 90-second function timeout C) The 50-second ruleset load sits outside any durable operation — so it is not checkpointed, and it re-executes on every single resume; each resume pays 50 seconds of S3 load plus replay of a growing checkpoint log before reaching any new work, and after enough review cycles a resume can no longer finish inside 90 seconds — it is killed mid-replay, retried, and killed again D) The checkpoint log hit its per-execution size limit and Lambda silently stopped scheduling replays — but exceeding durable execution limits surfaces as explicit errors in the execution history, not a silent stall, and the 90-second duration signature points at the invocation clock Answer in the comments. #AWS #Serverless #SystemDesign #DistributedSystems #Lambda ## Community discussion Top comments from developers on daily.dev. **@joudawad** · 3 upvotes > The answer is C. > > Replay is the whole trick behind durable functions _(checkpoint-and-replay — the handler re-runs from the top on every resume; completed steps and waits return their stored results instead of re-executing)_, and it has one iron rule: only durable operations are skipped. `context.step()` results are checkpointed. `context.wait()` completions are checkpointed. Everything else in your handler is just code — and code runs again, from the top, on every resume. > > This workflow resumes a lot. Every 6-hour review cycle ends a wait, which triggers a fresh invocation. Each of those... **@joudawad** · 2 upvotes > B — `context.wait()` is the opposite of running: the execution checkpoints, suspends, and stops billing entirely. When the wait elapses, Lambda starts a new invocation with a fresh function timeout. Waits cost nothing here — the resumes were the problem. **@joudawad** · 2 upvotes > D — Durable executions do have per-execution limits on operations and checkpointed state, and hitting them matters at high step counts. But limit violations surface as errors in the execution history. A silent stall with duration pinned at the function timeout is the signature of an invocation clock, not a quota. **@joudawad** · 2 upvotes > A — The execution timeout and the function timeout are different clocks. The execution timeout (up to 1 year) bounds the entire durable execution's lifetime; the function timeout bounds each individual invocation. These executions were days from their execution deadline. They were dying 90 seconds at a time. **@joudawad** · 2 upvotes > Also, it would mean a lot to me if you could support my content and stay in touch 🙏 > > - YouTube: [https://www.youtube.com/@system-design-lab](https://www.youtube.com/@system-design-lab) > - LinkedIn: [https://www.linkedin.com/in/joud-awad/](https://www.linkedin.com/in/joud-awad/) > - Medium Blog: [https://joudwawad.medium.com/](https://joudwawad.medium.com/) > - Substack: [https://joudawad.substack.com/](https://joudawad.substack.com/) ## Similar posts on daily.dev - [Weighing the benefits of AWS Lambda’s durable functions](https://daily.dev/posts/weighing-the-benefits-of-aws-lambda-s-durable-functions-qmmbps0ur) · InfoWorld · 0 upvotes · 0 comments - [AWS Lambda Durable Execution SDK for Java now available in Developer Preview](https://daily.dev/posts/aws-lambda-durable-execution-sdk-for-java-now-available-in-developer-preview-stqfaqm81) · AWS · 0 upvotes · 0 comments - [AWS re:Invent 2025: Why the future of the cloud is durable](https://daily.dev/posts/aws-re-invent-2025-why-the-future-of-the-cloud-is-durable-0moyrj2lz) · Temporal · 1 upvotes · 0 comments - [Scaling a Financial Reconciliation Pipeline With Serverless](https://daily.dev/posts/scaling-a-financial-reconciliation-pipeline-with-serverless-gynvqfhrg) · Cloud Native Now · 0 upvotes · 0 comments --- Tags: [#career](https://daily.dev/tags/career), [#serverless](https://daily.dev/tags/serverless), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#aws-lambda](https://daily.dev/tags/aws-lambda) [View this post on daily.dev](https://daily.dev/posts/day-9-30-aws-system-design-patterns-ovlngbuvs) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/day-9-30-aws-system-design-patterns-ovlngbuvs","headline":"Day 9/30 AWS System Design Patterns","text":"A scenario-based AWS system design puzzle involving a Lambda durable function powering a loan approval workflow. The workflow downloads a 40 MB compliance ruleset from S3 outside any checkpointed step, causing it to re-execute on every replay. As the checkpoint log grows with each review cycle, the 50-second S3 load plus replay overhead eventually exceeds the 90-second Lambda function timeout, freezing 1,900 applications mid-review. The correct diagnosis (option C) highlights a classic durable execution pitfall: non-idempotent or expensive code placed outside durable steps is not checkpointed and re-runs on every resume, compounding cost with each replay.","url":"https://daily.dev/posts/day-9-30-aws-system-design-patterns-ovlngbuvs","datePublished":"2026-07-29T17:44:09.346Z","dateModified":"2026-09-13T21:19:51.126Z","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0","description":"Principal Solution Architecture ","worksFor":{"@type":"Organization","name":"Metalab","logo":"https://www.google.com/s2/favicons?domain=metalab.com&sz=128"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":84960}},"image":"https://media.daily.dev/image/upload/s--vfVZSwnH--/f_auto/v1785347052/posts/oVLngbUvs?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":83},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":15}],"comment":[{"@type":"Comment","text":"The answer is C.\nReplay is the whole trick behind durable functions (checkpoint-and-replay — the handler re-runs from the top on every resume; completed steps and waits return their stored results instead of re-executing), and it has one iron rule: only durable operations are skipped. context.step() results are checkpointed. context.wait() completions are checkpointed. Everything else in your handler is just code — and code runs again, from the top, on every resume.\nThis workflow resumes a lot. Every 6-hour review cycle ends a wait, which triggers a fresh invocation. Each of those invocations starts at line 1: download 40 MB from S3 (object storage), parse for ~50 seconds, then replay the checkpoint log — validate (skipped), credit report (skipped), wait 1 (skipped), check 1 (skipped), wait 2 (skipped)… The replay itself is fast per checkpoint, but the log grows with every cycle. By review cycle 8 or 9, 50 seconds of un-checkpointed loading plus replay overhead plus the actual status check no longer fit inside the 90-second function timeout. The invocation is killed mid-flight.\nHere is the vicious part: a timed-out resume doesn’t fail the durable execution — the platform retries the resume. The retry starts at line 1, pays the same 50-second tax, and dies the same death. The execution is alive, unexpired, and permanently unable to make progress. No business exception is ever thrown, because the business logic never gets reached. The only witness is Duration pinned at the timeout.\nTwo fixes:\nFix 1 — Move the heavy work into the world replay can skip. Load the ruleset inside a context.step() that returns only the small values the workflow actually needs downstream (thresholds, version id) — checkpointed once, skipped on every replay. If steps genuinely need the full parsed object, cache it at module scope keyed on the ruleset version: warm environments reuse it, and a cold resume pays the load once inside a step boundary, not on every replay of history.\nFix 2 — Design for replay speed as history grows. Keep the code path outside steps trivial and deterministic (that is also the correctness rule — no clocks, randomness, or I/O outside durable operations, or replay diverges). Use one step per side effect rather than dozens of fine-grained checkpoints, and set the function timeout with headroom for the longest replay your workflow can accumulate — a durable function’s resume cost is a function of its history, not just its next action.\nThe general pattern is worth naming: this is durable execution, the same model as Temporal or Azure Durable Functions, and it always carries this contract — replay must be fast and deterministic, and anything expensive or side-effectful belongs inside a checkpointed activity. The 15-minute wall came down; the replay discipline came with it.","datePublished":"2026-07-29T17:44:34.191Z","url":"https://daily.dev/posts/oVLngbUvs#c-b6eNBMKP0","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3}},{"@type":"Comment","text":"B — context.wait() is the opposite of running: the execution checkpoints, suspends, and stops billing entirely. When the wait elapses, Lambda starts a new invocation with a fresh function timeout. Waits cost nothing here — the resumes were the problem.","datePublished":"2026-07-29T17:44:48.377Z","url":"https://daily.dev/posts/oVLngbUvs#c-shiPF1pFe","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"D — Durable executions do have per-execution limits on operations and checkpointed state, and hitting them matters at high step counts. But limit violations surface as errors in the execution history. A silent stall with duration pinned at the function timeout is the signature of an invocation clock, not a quota.","datePublished":"2026-07-29T17:44:54.953Z","url":"https://daily.dev/posts/oVLngbUvs#c-xFkhWYcs2","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"A — The execution timeout and the function timeout are different clocks. The execution timeout (up to 1 year) bounds the entire durable execution’s lifetime; the function timeout bounds each individual invocation. These executions were days from their execution deadline. They were dying 90 seconds at a time.","datePublished":"2026-07-29T17:44:45.187Z","url":"https://daily.dev/posts/oVLngbUvs#c-khnfVdM5I","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"Also, it would mean a lot to me if you could support my content and stay in touch 🙏\n\nYouTube: https://www.youtube.com/@system-design-lab\nLinkedIn: https://www.linkedin.com/in/joud-awad/\nMedium Blog: https://joudwawad.medium.com/\nSubstack: https://joudawad.substack.com/","datePublished":"2026-07-29T17:45:08.616Z","url":"https://daily.dev/posts/oVLngbUvs#c-ko1Vr8l8I","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh","name":"Joud Awad"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Joud Awad","item":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh"},{"@type":"ListItem","position":3,"name":"Day 9/30 AWS System Design Patterns"}]} ```