--- title: Why Your Agent Disagrees With Itself (And What To Do... description: AI agents often produce inconsistent outputs for the same input, not due to model flaws but because ambiguous data points near decision boundaries lack... canonical: https://daily.dev/posts/why-your-agent-disagrees-with-itself-and-what-to-do-about-it---diane-lin-datadog-inwpuj17i twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog | daily.dev og:description: AI agents often produce inconsistent outputs for the same input, not due to model flaws but because ambiguous data points near decision boundaries lack... og:url: https://daily.dev/posts/why-your-agent-disagrees-with-itself-and-what-to-do-about-it---diane-lin-datadog-inwpuj17i og:image: https://api.daily.dev/og/posts/inwpuJ17I.png og:image:alt: Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog **[AI Engineer](https://daily.dev/sources/aidotengineer)** · 25 min read · 0 upvotes · 0 comments ## Summary AI agents often produce inconsistent outputs for the same input, not due to model flaws but because ambiguous data points near decision boundaries lack sufficient context. Using active learning principles, specifically query-by-committee (running the model multiple times and flagging disagreements), developers can identify these gray-zone cases efficiently. Rather than expensive fine-tuning, augmenting agents with semantic memory (explicit domain rules and policies) and episodic memory (past labeled similar cases) can resolve most inconsistencies automatically. Experimental results on 93 cybersecurity alerts showed this approach reduced flip-flopping from 25% to 10%, with the remaining cases routed to human review for further disambiguation. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does the same LLM give different answers when I run it multiple times on the exact same input? Inconsistent outputs typically occur for data points near the decision boundary where the task itself is genuinely ambiguous, not because of random model failure. Even human experts disagree on these borderline cases, since the correct label often depends on unstated context or policy preferences. The disagreement signal from multiple runs or models is more reliable than the model's own confidence score for finding these cases. _Teams debugging flaky agent outputs can find grounded engineering approaches like this one on daily.dev._ ### How can I reduce inconsistent classifications from an AI agent without fine-tuning it? Augmenting the agent with semantic memory (explicit domain policy knowledge, such as company-specific rules) and episodic memory (past similar labeled cases) can reduce inconsistency without costly fine-tuning. In one test on 93 cybersecurity alerts run three times each, 25% flip-flopped originally; episodic memory alone made 15% of those consistent, leaving 10% for human review to add clarifying knowledge. _Anyone weighing fine-tuning against lighter-weight fixes can track approaches like memory augmentation on daily.dev._ ### What is query by committee and how is it used to find unreliable AI agent predictions? Query by committee identifies problematic data points by running several different models, or the same model multiple times, and flagging cases where the outputs disagree. This disagreement signal reveals where a model is genuinely uncertain, unlike raw confidence scores from language models, which are unreliable because the model often does not know what it does not know. _Developers building active-learning pipelines for agents can follow techniques like this via daily.dev._ ## Similar posts on daily.dev - [Why most AI agents disappoint in production \(and what to fix first\)](https://daily.dev/posts/why-most-ai-agents-disappoint-in-production-and-what-to-fix-first--vwp99nhc8) · InfoWorld · 0 upvotes · 0 comments - [Your agent doesn't have a model problem](https://daily.dev/posts/your-agent-doesn-t-have-a-model-problem-3snlafbtz) · Engineering Enablement · 1 upvotes · 1 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#cyber](https://daily.dev/tags/cyber), [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents) [View this post on daily.dev](https://daily.dev/posts/why-your-agent-disagrees-with-itself-and-what-to-do-about-it---diane-lin-datadog-inwpuj17i) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog","url":"https://daily.dev/posts/why-your-agent-disagrees-with-itself-and-what-to-do-about-it---diane-lin-datadog-inwpuj17i","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/why-your-agent-disagrees-with-itself-and-what-to-do-about-it---diane-lin-datadog-inwpuj17i"},"datePublished":"2026-07-20T06:39:37.561Z","dateModified":"2026-09-13T20:05:12.475Z","description":"AI agents often produce inconsistent outputs for the same input, not due to model flaws but because ambiguous data points near decision boundaries lack...","image":"https://i.ytimg.com/vi/wEc9aG7cRQc/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/wEc9aG7cRQc/sddefault.jpg","isAccessibleForFree":true,"articleSection":"AI Engineer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AI Engineer","logo":"https://media.daily.dev/image/upload/s--u5PucxNT--/f_auto/v1724338940/logos/aidotengineer","url":"https://daily.dev/sources/aidotengineer"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/why-your-agent-disagrees-with-itself-and-what-to-do-about-it---diane-lin-datadog-inwpuj17i","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,cyber,llm,ai-agents","timeRequired":"PT25M","video":{"@type":"VideoObject","name":"Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog","description":"AI agents often produce inconsistent outputs for the same input, not due to model flaws but because ambiguous data points near decision boundaries lack...","thumbnailUrl":"https://i.ytimg.com/vi/wEc9aG7cRQc/sddefault.jpg","uploadDate":"2026-07-20T06:39:37.561Z","duration":"PT25M","url":"https://api.daily.dev/r/inwpuJ17I","embedUrl":"https://www.youtube.com/embed/wEc9aG7cRQc"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI Engineer","item":"https://daily.dev/sources/aidotengineer"},{"@type":"ListItem","position":3,"name":"Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/why-your-agent-disagrees-with-itself-and-what-to-do-about-it---diane-lin-datadog-inwpuj17i#faq","mainEntity":[{"@type":"Question","name":"Why does the same LLM give different answers when I run it multiple times on the exact same input?","acceptedAnswer":{"@type":"Answer","text":"Inconsistent outputs typically occur for data points near the decision boundary where the task itself is genuinely ambiguous, not because of random model failure. Even human experts disagree on these borderline cases, since the correct label often depends on unstated context or policy preferences. The disagreement signal from multiple runs or models is more reliable than the model's own confidence score for finding these cases. Teams debugging flaky agent outputs can find grounded engineering approaches like this one on daily.dev."}},{"@type":"Question","name":"How can I reduce inconsistent classifications from an AI agent without fine-tuning it?","acceptedAnswer":{"@type":"Answer","text":"Augmenting the agent with semantic memory (explicit domain policy knowledge, such as company-specific rules) and episodic memory (past similar labeled cases) can reduce inconsistency without costly fine-tuning. In one test on 93 cybersecurity alerts run three times each, 25% flip-flopped originally; episodic memory alone made 15% of those consistent, leaving 10% for human review to add clarifying knowledge. Anyone weighing fine-tuning against lighter-weight fixes can track approaches like memory augmentation on daily.dev."}},{"@type":"Question","name":"What is query by committee and how is it used to find unreliable AI agent predictions?","acceptedAnswer":{"@type":"Answer","text":"Query by committee identifies problematic data points by running several different models, or the same model multiple times, and flagging cases where the outputs disagree. This disagreement signal reveals where a model is genuinely uncertain, unlike raw confidence scores from language models, which are unreliable because the model often does not know what it does not know. Developers building active-learning pipelines for agents can follow techniques like this via daily.dev."}}]} ```