--- title: All your workflows are about to be long running. Build... description: Drawing on a real nightly pipeline that processed 100,000 patient records and repeatedly died mid-run, the piece argues that long-running work (batch jobs,... canonical: https://daily.dev/posts/all-your-workflows-are-about-to-be-long-running-build-for-it--ff8epi154 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: All your workflows are about to be long running. Build for it. | daily.dev og:description: Drawing on a real nightly pipeline that processed 100,000 patient records and repeatedly died mid-run, the piece argues that long-running work (batch jobs,... og:url: https://daily.dev/posts/all-your-workflows-are-about-to-be-long-running-build-for-it--ff8epi154 og:image: https://api.daily.dev/og/posts/Ff8Epi154.png og:image:alt: All your workflows are about to be long running. Build for it. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # All your workflows are about to be long running. Build for it. **[Inngest Blog](https://daily.dev/sources/inngest)** · 7 min read · 0 upvotes · 0 comments ## Summary Drawing on a real nightly pipeline that processed 100,000 patient records and repeatedly died mid-run, the piece argues that long-running work (batch jobs, agent loops, webhook-driven approvals) must be architected for mid-execution failure rather than relying on better retry logic. Key principles: commit progress in small, memoized steps with idempotency keys for side effects; keep state outside the worker process so any healthy worker can resume; make waiting free by releasing compute instead of holding a thread or slot; fan out massive workloads into isolated chunks rather than one giant job; and keep individual steps short while allowing overall runs to span long durations. The author describes migrating this pipeline to Inngest's durable execution primitives (steps, step.sleep, flow control, versioning mid-run), which eliminated hand-rolled concurrency and retry code and turned nightly failures into reliably finished runs. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I make a long-running batch pipeline resume from where it failed instead of starting over? Split the work into small steps and save each step's result to durable storage outside the worker process. On retry, completed steps return their saved result instead of re-running, turning a restart into a resume. Steps with side effects also need an idempotency key, so a step that already charged a card or sent a record doesn't repeat that action on retry. _Developers designing resumable pipelines compare durable execution approaches like this one on daily.dev._ ### Why does a paused or waiting job still consume resources in a hand-built queue system? Hand-rolled systems typically keep a worker thread, connection, or slot occupied for the entire time a job is waiting on a webhook, human approval, or external event, even though nothing is actively happening. Enough paused jobs holding slots can stall the whole queue. The fix is a wait mechanism that releases compute entirely and resumes exactly where it paused once the signal arrives. _Teams debugging stalled queues can find architecture patterns like this on daily.dev before redesigning their wait logic._ ### Why should a million-record batch job be split into many smaller runs instead of one big loop? A single giant run that loops over every record means one bad record or one crashed pod can take down the entire dataset's progress, and the whole dataset must sit in memory at once. Fanning out into per-chunk runs isolates failures so a bad record or dead pod only affects its own chunk, avoids out-of-memory failures by design, and lets flow control govern how fast chunks hit downstream systems. _Engineers planning fan-out architectures for large batch workloads can track these patterns on daily.dev._ ## Similar posts on daily.dev - [How to Build AI Agents That Don’t Start Over When They Fail](https://daily.dev/posts/how-to-build-ai-agents-that-don-t-start-over-when-they-fail-hdtnqpnlp) · System Design Newsletter · 3 upvotes · 1 comments - [What dead f\*cking last taught me about durable execution](https://daily.dev/posts/what-dead-f-cking-last-taught-me-about-durable-execution-yk6qac90n) · Inngest Blog · 0 upvotes · 0 comments - [Building Reliable Production AI with Durable Workflows](https://daily.dev/posts/building-reliable-production-ai-with-durable-workflows-8k7gdooen) · Salesforce Engineering · 2 upvotes · 1 comments - [Durable AI Workflows 2026: Inngest, Trigger.dev, Vercel Workflow](https://daily.dev/posts/durable-ai-workflows-2026-inngest-trigger-dev-vercel-workflow-un2tgyyqs) · Alex CloudStar · 0 upvotes · 0 comments - [5 Resilience Patterns for AI Agents](https://daily.dev/posts/5-resilience-patterns-for-ai-agents-oiijblpah) · The T-Shaped Dev · 1 upvotes · 0 comments --- Tags: [#golang](https://daily.dev/tags/golang), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#workflow-orchestration](https://daily.dev/tags/workflow-orchestration), [#inngest](https://daily.dev/tags/inngest) [View this post on daily.dev](https://daily.dev/posts/all-your-workflows-are-about-to-be-long-running-build-for-it--ff8epi154) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"All your workflows are about to be long running. Build for it.","url":"https://daily.dev/posts/all-your-workflows-are-about-to-be-long-running-build-for-it--ff8epi154","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/all-your-workflows-are-about-to-be-long-running-build-for-it--ff8epi154"},"datePublished":"2026-10-01T17:25:03.467Z","dateModified":"2026-10-01T17:34:46.429Z","description":"Drawing on a real nightly pipeline that processed 100,000 patient records and repeatedly died mid-run, the piece argues that long-running work (batch jobs,...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ce5aada1a42cfbcb1f8187e1966b8fa6?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ce5aada1a42cfbcb1f8187e1966b8fa6?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Inngest Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Inngest Blog","logo":"https://media.daily.dev/image/upload/s--ieZKUa5y--/c_limit,w_256/f_auto,q_auto/v1789329423/logos/inngest?_a=BAMAMicg0","url":"https://daily.dev/sources/inngest"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/all-your-workflows-are-about-to-be-long-running-build-for-it--ff8epi154","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"golang,distributed-systems,workflow-orchestration,inngest","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Inngest Blog","item":"https://daily.dev/sources/inngest"},{"@type":"ListItem","position":3,"name":"All your workflows are about to be long running. Build for it."}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/all-your-workflows-are-about-to-be-long-running-build-for-it--ff8epi154#faq","mainEntity":[{"@type":"Question","name":"How do I make a long-running batch pipeline resume from where it failed instead of starting over?","acceptedAnswer":{"@type":"Answer","text":"Split the work into small steps and save each step's result to durable storage outside the worker process. On retry, completed steps return their saved result instead of re-running, turning a restart into a resume. Steps with side effects also need an idempotency key, so a step that already charged a card or sent a record doesn't repeat that action on retry. Developers designing resumable pipelines compare durable execution approaches like this one on daily.dev."}},{"@type":"Question","name":"Why does a paused or waiting job still consume resources in a hand-built queue system?","acceptedAnswer":{"@type":"Answer","text":"Hand-rolled systems typically keep a worker thread, connection, or slot occupied for the entire time a job is waiting on a webhook, human approval, or external event, even though nothing is actively happening. Enough paused jobs holding slots can stall the whole queue. The fix is a wait mechanism that releases compute entirely and resumes exactly where it paused once the signal arrives. Teams debugging stalled queues can find architecture patterns like this on daily.dev before redesigning their wait logic."}},{"@type":"Question","name":"Why should a million-record batch job be split into many smaller runs instead of one big loop?","acceptedAnswer":{"@type":"Answer","text":"A single giant run that loops over every record means one bad record or one crashed pod can take down the entire dataset's progress, and the whole dataset must sit in memory at once. Fanning out into per-chunk runs isolates failures so a bad record or dead pod only affects its own chunk, avoids out-of-memory failures by design, and lets flow control govern how fast chunks hit downstream systems. Engineers planning fan-out architectures for large batch workloads can track these patterns on daily.dev."}}]} ```