--- title: Building a durable /goal agent loop that survives crashes description: A durable /goal agent loop, built on Inngest, can survive worker crashes without losing progress or re-running finished work. Killing the worker with kill -9... canonical: https://daily.dev/posts/building-a-durable-goal-agent-loop-that-survives-crashes-gwywefazz twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Building a durable /goal agent loop that survives crashes | daily.dev og:description: A durable /goal agent loop, built on Inngest, can survive worker crashes without losing progress or re-running finished work. Killing the worker with kill -9... og:url: https://daily.dev/posts/building-a-durable-goal-agent-loop-that-survives-crashes-gwywefazz og:image: https://api.daily.dev/og/posts/GwyWefaZZ.png og:image:alt: Building a durable /goal agent loop that survives crashes og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Building a durable /goal agent loop that survives crashes **[Inngest Blog](https://daily.dev/sources/inngest)** · 19 min read · 0 upvotes · 0 comments ## Summary A durable /goal agent loop, built on Inngest, can survive worker crashes without losing progress or re-running finished work. Killing the worker with kill -9 mid-attempt let a new worker resume about 15 seconds later, with every finished turn preserved exactly once. The key boundary found: state survives a dead worker indefinitely, but the run itself only survives as long as the function's retry budget, fixed by raising retries from the default (a few minutes) to 20 (hours). Risky agent-generated code runs only inside a disposable sandbox, never on the worker, after a plain child-process timeout failed to stop a runaway while(true) loop. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What's the difference between state durability and run durability in Inngest's durable execution model? Step state survives a dead worker indefinitely, but the overall run only survives as long as the next function that must execute keeps retrying. If a worker stays down past the function's retry window, the run fails even though every finished step's results are still saved; raising the function's retries (e.g. to 20) widens that window from minutes to hours. _Anyone designing long-running agent workflows can study durable execution trade-offs like this on daily.dev._ ### How do you safely run untrusted AI-generated code that might hang or crash? Run only the code-execution step inside a disposable sandbox, never on the worker that drives the loop, because a plain child-process timeout isn't enough; killing a wrapper process can still leave a child spinning on an infinite loop and pinning a core. Reading, editing and typechecking source stays on the worker since those never execute the code, and only the grader step touches the sandbox, which has no network access. _Developers isolating AI-generated code execution can track patterns like this via daily.dev._ ### How do you detect when an AI agent is overfitting to its own evaluation checks? Track the generalization gap between train and holdout scores: a loop that learns to satisfy its own check report shows a train score that the holdout score doesn't match. Holdout cases are scored only once, at the end, on the final best attempt, and are never shown to the agent during attempts, keeping the gap measurement clean. _Teams tuning agent evaluation setups can follow techniques like this on daily.dev._ ## Similar posts on daily.dev - [How to Build AI Agents That Don’t Start Over When They Fail](https://daily.dev/posts/how-to-build-ai-agents-that-don-t-start-over-when-they-fail-hdtnqpnlp) · System Design Newsletter · 3 upvotes · 1 comments - [Why raising your serverless timeout won't save long jobs](https://daily.dev/posts/why-raising-your-serverless-timeout-won-t-save-long-jobs-qdgq0mach) · Inngest Blog · 0 upvotes · 0 comments - [How to build agents that survive crashes with Pi Durable](https://daily.dev/posts/how-to-build-agents-that-survive-crashes-with-pi-durable-zt8jcyleg) · Flavio Copes · 8 upvotes · 0 comments - [Designing a Production-Ready Agent Harness With Persistence and Checkpointing](https://daily.dev/posts/designing-a-production-ready-agent-harness-with-persistence-and-checkpointing-4ygffqjsx) · SitePoint · 1 upvotes · 0 comments - [All your workflows are about to be long running. Build for it.](https://daily.dev/posts/all-your-workflows-are-about-to-be-long-running-build-for-it--ff8epi154) · Inngest Blog · 0 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#typescript](https://daily.dev/tags/typescript), [#inngest](https://daily.dev/tags/inngest) [View this post on daily.dev](https://daily.dev/posts/building-a-durable-goal-agent-loop-that-survives-crashes-gwywefazz) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Building a durable /goal agent loop that survives crashes","url":"https://daily.dev/posts/building-a-durable-goal-agent-loop-that-survives-crashes-gwywefazz","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/building-a-durable-goal-agent-loop-that-survives-crashes-gwywefazz"},"datePublished":"2026-11-02T16:21:50.802Z","dateModified":"2026-11-02T16:22:21.936Z","description":"A durable /goal agent loop, built on Inngest, can survive worker crashes without losing progress or re-running finished work. Killing the worker with kill -9...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9c483cae29f3e0b0e5ced30777733791?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9c483cae29f3e0b0e5ced30777733791?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Inngest Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Inngest Blog","logo":"https://media.daily.dev/image/upload/s--ieZKUa5y--/c_limit,w_256/f_auto,q_auto/v1789329423/logos/inngest?_a=BAMAMicg0","url":"https://daily.dev/sources/inngest"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/building-a-durable-goal-agent-loop-that-survives-crashes-gwywefazz","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,typescript,inngest","timeRequired":"PT19M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Inngest Blog","item":"https://daily.dev/sources/inngest"},{"@type":"ListItem","position":3,"name":"Building a durable /goal agent loop that survives crashes"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/building-a-durable-goal-agent-loop-that-survives-crashes-gwywefazz#faq","mainEntity":[{"@type":"Question","name":"What's the difference between state durability and run durability in Inngest's durable execution model?","acceptedAnswer":{"@type":"Answer","text":"Step state survives a dead worker indefinitely, but the overall run only survives as long as the next function that must execute keeps retrying. If a worker stays down past the function's retry window, the run fails even though every finished step's results are still saved; raising the function's retries (e.g. to 20) widens that window from minutes to hours. Anyone designing long-running agent workflows can study durable execution trade-offs like this on daily.dev."}},{"@type":"Question","name":"How do you safely run untrusted AI-generated code that might hang or crash?","acceptedAnswer":{"@type":"Answer","text":"Run only the code-execution step inside a disposable sandbox, never on the worker that drives the loop, because a plain child-process timeout isn't enough; killing a wrapper process can still leave a child spinning on an infinite loop and pinning a core. Reading, editing and typechecking source stays on the worker since those never execute the code, and only the grader step touches the sandbox, which has no network access. Developers isolating AI-generated code execution can track patterns like this via daily.dev."}},{"@type":"Question","name":"How do you detect when an AI agent is overfitting to its own evaluation checks?","acceptedAnswer":{"@type":"Answer","text":"Track the generalization gap between train and holdout scores: a loop that learns to satisfy its own check report shows a train score that the holdout score doesn't match. Holdout cases are scored only once, at the end, on the final best attempt, and are never shown to the agent during attempts, keeping the gap measurement clean. Teams tuning agent evaluation setups can follow techniques like this on daily.dev."}}]} ```