--- title: DigitalOcean adds Signals and Simulations to catch agent... description: DigitalOcean launched two Public Preview features under Agent Learning Services: Signals, which monitors production traffic on Managed Agents and Serverless... canonical: https://daily.dev/posts/digitalocean-adds-signals-and-simulations-to-catch-agent-failures-in-production-and-test-fixes-befor-t3417ua9u twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: DigitalOcean adds Signals and Simulations to catch agent failures in production and test fixes before shipping | daily.dev og:description: DigitalOcean launched two Public Preview features under Agent Learning Services: Signals, which monitors production traffic on Managed Agents and Serverless... og:url: https://daily.dev/posts/digitalocean-adds-signals-and-simulations-to-catch-agent-failures-in-production-and-test-fixes-befor-t3417ua9u og:image: https://api.daily.dev/og/posts/T3417uA9u.png og:image:alt: DigitalOcean adds Signals and Simulations to catch agent failures in production and test fixes before shipping og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # DigitalOcean adds Signals and Simulations to catch agent failures in production and test fixes before shipping **[Collections](https://daily.dev/sources/collections)** · 3 min read · 1 upvotes · 0 comments ## Summary DigitalOcean launched two Public Preview features under Agent Learning Services: Signals, which monitors production traffic on Managed Agents and Serverless Inference to flag failing sessions (frustrated users, failing tool calls, infra problems) without extra instrumentation and turns them into evals with one click, and Simulations, part of DigitalOcean Evaluations, which builds multi-turn test scenarios from real production sessions and runs candidate fixes in an isolated sandbox using the same tools and harness as production, scored by an LLM judge on metrics like goal completion and tool-use accuracy. DigitalOcean's paper reports 82% informativeness and a 1.52x efficiency gain over random sampling on τ-bench. Supported harnesses are Claude Code, Codex, and OpenCode; LangChain and CrewAI are not yet supported. ## Content Agent failures often pass every health check. A tool call hits the wrong database region, returns a 200, and the user gets a slower, worse answer. Nothing alerts. The logs look fine, and weeks later someone discovers hundreds of users were affected. DigitalOcean is targeting that gap with two new capabilities under Agent Learning Services: Signals and Simulations. Both are in Public Preview. ## Signals Signals watches production traffic on Managed Agents and Serverless Inference and flags sessions that went wrong. It looks for frustrated users, failing tool calls and infrastructure problems, with no extra instrumentation. A typical catch is a tool call that succeeded technically but queried the wrong region. It doesn't use an LLM to grade every conversation. It relies on cheap rules that look for signs of human trouble: a user repeating themselves, asking for a human, or a tool being retried in a loop. An accompanying paper describes the triage method. On the τ-bench benchmark it reached an 82% informativeness rate and a 1.52x efficiency gain over random sampling. A flagged session can be turned into an eval with one click. ## Simulations Simulations is part of DigitalOcean Evaluations. It tests a change against realistic multi-turn scenarios built from real production sessions or from the agent's configuration, before anything is deployed. Candidates run in an isolated sandbox with the same tools and harness as production. A simulated user plays the other side of the conversation, and an LLM judge scores the full trajectory on metrics like goal completion and tool-use accuracy. Because a flagged session can be replayed with the proposed fix, you can test against the exact conversation that failed. The workflow is: Signals flags the session, Simulations tests the fix, a human approves, and the bug dies at user one instead of user 400. ## Other pieces DigitalOcean also launched Insights in public preview, and Agent Data Services in private preview. Agent Data Services lets an agent review old support call recordings and documents to work out what caused a problem, pulling scattered clues into one place. ## Why this matters Teams have mostly had three options: pay an LLM to grade every conversation, read transcripts by hand, or invent test cases from imagination. The most important failures only show up in production and rarely make it into a test set. Here, real production failures become the test set automatically. That loop is the part I find convincing. ## Limits The preview supports harnesses like Claude Code, Codex and OpenCode. Third-party frameworks such as LangChain and CrewAI aren't supported yet. ## Questions this post answers ### What do DigitalOcean's new Signals and Simulations features do for AI agents? Signals monitors production traffic on Managed Agents and Serverless Inference to automatically flag failing sessions, such as frustrated users, failing tool calls, or infrastructure problems, without needing extra instrumentation, and these can become evals with one click. Simulations then lets you run a candidate fix against multi-turn scenarios built from those real sessions in an isolated sandbox using the same tools and harness as production, scored by an LLM judge on metrics like goal completion. _Teams hardening production agents can follow how evaluation tooling like this evolves on daily.dev._ ### Which agent harnesses does DigitalOcean's Simulations feature support? Simulations supports Claude Code, Codex, and OpenCode as of its Public Preview launch. Third-party agent frameworks like LangChain and CrewAI are not yet supported, so teams built on those frameworks cannot use it until support is added. _Developers choosing an agent framework can track harness compatibility updates like this on daily.dev._ ### How effective is DigitalOcean's Signals triage method at surfacing useful failing sessions compared to random sampling? DigitalOcean's accompanying paper reports an 82% informativeness rate and a 1.52x efficiency gain over random sampling on the τ-bench benchmark, meaning the triage method surfaces useful failing sessions more reliably than picking sessions at random, while requiring no additional instrumentation on the monitored agent. _Anyone building eval pipelines for production agents can weigh benchmark results like these on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 4 discussions and 27 comments across x (as of 2026-10-13). **TL;DR:** Reaction is largely enthusiastic about turning real production failures into replayable tests, with the silent-failure problem (agents reporting success when they actually failed) resonating strongly; a few people push for deeper verification of tool results rather than trusting the agent's own claims. **Sentiment:** 65% positive · 30% mixed · 5% skeptical **The case for** - The idea of replaying a broken session against a proposed fix before shipping is seen as filling a real gap in agent debugging. - Catching silent failures (like a successful-looking 200 response that actually did the wrong thing) before many users are affected is viewed as more valuable than model upgrades. - Turning real-world failures into repeatable tests is praised as a smart, correct feedback loop. **The pushback** - Some want verification of actual tool results rather than just trusting what the agent claims it did, citing the refund example where the agent reported a different amount than what was actually processed. - Concern that most users who hit a failure won't report it, they just quietly leave, making detection harder than the tool implies. - A question remains about whether traces can show which specific checks failed without retaining users' raw data. **By community** - x (positive): Replies are mostly enthusiastic about the silent-failure detection and replay-to-fix workflow, with a handful of more substantive requests for stronger tool-result verification. **Open questions** - Can the system show which specific verification checks failed without retaining users' raw/sensitive data? - How well does the approach handle failures that users never report and simply abandon silently? **Highlights** > @testingcatalog The production-failure-to-simulation workflow looks useful. One thing I'd want from these tests is verification of actual tool results, not just what the agent claims it did. DigitalOcean's own refund example showed an agent reporting $40 refunded when the tool had processed just > — [vermaizer on x](https://x.com/vermaizer/status/2109671567731089604) > @testingcatalog A tool call that succeeds but queries the wrong region returns a 200, so no error alert ever fires. Flagging it from the session itself, then replaying that exact conversation against the fix, means you verify on the case that actually broke. > — [DoDataThings on x](https://x.com/DoDataThings/status/2109685556255047702) > @rohanpaul_ai most users wont report a polite 'try again later', they just leave quietly :( > — [dusangran on x · 2 points](https://x.com/dusangran/status/2109682600990118232) > @rohanpaul_ai A 200 only proves the request was handled. I'd check the result against the task: correct region, expected record, and usable freshness. Can the trace show which of those checks failed without retaining the user's raw data? > — [cliboub on x](https://x.com/cliboub/status/2109685119103414408) > @testingcatalog in DigitalOcean's testing it replayed 10 production scenarios end-to-end in under a minute, same agent setup as prod, mock tools for costly actions. picture the on-call engineer approving a fix for one angry ticket before the next customer finishes typing "is anyone there." > — [delx369 on x](https://x.com/delx369/status/2109698711135900144) **Source threads** - [x](https://x.com/testingcatalog/status/2109667399142093203) · 1 points · 9 comments - [x](https://x.com/rohanpaul_ai/status/2109681008484467076) · 0 points · 8 comments - [x](https://x.com/trikcode/status/2109669720882364664) · 0 points · 10 comments - [x](https://x.com/trikcode/status/2109668611124027605) · 0 points · 0 comments ## Similar posts on daily.dev - [Introducing DigitalOcean Insights: Built-in Observability for Your Agents, Applications, and Infrastructure](https://daily.dev/posts/introducing-digitalocean-insights-built-in-observability-for-your-agents-applications-and-infrast-9qs5w4b2y) · DigitalOcean · 1 upvotes · 0 comments - [Introducing DigitalOcean Agent Data Services: The Data Foundation for Agents](https://daily.dev/posts/introducing-digitalocean-agent-data-services-the-data-foundation-for-agents-yyuid9bbr) · DigitalOcean · 1 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#observability](https://daily.dev/tags/observability), [#claude-code](https://daily.dev/tags/claude-code), [#digitalocean](https://daily.dev/tags/digitalocean) [View this post on daily.dev](https://daily.dev/posts/digitalocean-adds-signals-and-simulations-to-catch-agent-failures-in-production-and-test-fixes-befor-t3417ua9u) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"DigitalOcean adds Signals and Simulations to catch agent failures in production and test fixes before shipping","url":"https://daily.dev/posts/digitalocean-adds-signals-and-simulations-to-catch-agent-failures-in-production-and-test-fixes-befor-t3417ua9u","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/digitalocean-adds-signals-and-simulations-to-catch-agent-failures-in-production-and-test-fixes-befor-t3417ua9u"},"datePublished":"2026-10-12T15:24:25.652Z","dateModified":"2026-10-13T03:57:19.565Z","description":"DigitalOcean launched two Public Preview features under Agent Learning Services: Signals, which monitors production traffic on Managed Agents and Serverless...","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/digitalocean-adds-signals-and-simulations-to-catch-agent-failures-in-production-and-test-fixes-befor-t3417ua9u","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,observability,claude-code,digitalocean","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"DigitalOcean adds Signals and Simulations to catch agent failures in production and test fixes before shipping"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/digitalocean-adds-signals-and-simulations-to-catch-agent-failures-in-production-and-test-fixes-befor-t3417ua9u#faq","mainEntity":[{"@type":"Question","name":"What do DigitalOcean's new Signals and Simulations features do for AI agents?","acceptedAnswer":{"@type":"Answer","text":"Signals monitors production traffic on Managed Agents and Serverless Inference to automatically flag failing sessions, such as frustrated users, failing tool calls, or infrastructure problems, without needing extra instrumentation, and these can become evals with one click. Simulations then lets you run a candidate fix against multi-turn scenarios built from those real sessions in an isolated sandbox using the same tools and harness as production, scored by an LLM judge on metrics like goal completion. Teams hardening production agents can follow how evaluation tooling like this evolves on daily.dev."}},{"@type":"Question","name":"Which agent harnesses does DigitalOcean's Simulations feature support?","acceptedAnswer":{"@type":"Answer","text":"Simulations supports Claude Code, Codex, and OpenCode as of its Public Preview launch. Third-party agent frameworks like LangChain and CrewAI are not yet supported, so teams built on those frameworks cannot use it until support is added. Developers choosing an agent framework can track harness compatibility updates like this on daily.dev."}},{"@type":"Question","name":"How effective is DigitalOcean's Signals triage method at surfacing useful failing sessions compared to random sampling?","acceptedAnswer":{"@type":"Answer","text":"DigitalOcean's accompanying paper reports an 82% informativeness rate and a 1.52x efficiency gain over random sampling on the τ-bench benchmark, meaning the triage method surfaces useful failing sessions more reliably than picking sessions at random, while requiring no additional instrumentation on the monitored agent. Anyone building eval pipelines for production agents can weigh benchmark results like these on daily.dev."}}]} ```