--- title: Behavioral testing for AI agents | daily.dev description: Traditional unit tests can't catch behavioral failures in AI agents because they mock the LLM and test scaffolding, not model decisions. Behavioral testing... canonical: https://daily.dev/posts/behavioral-testing-for-ai-agents-9fwv9mpuy twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Behavioral testing for AI agents | daily.dev og:description: Traditional unit tests can't catch behavioral failures in AI agents because they mock the LLM and test scaffolding, not model decisions. Behavioral testing... og:url: https://daily.dev/posts/behavioral-testing-for-ai-agents-9fwv9mpuy og:image: https://api.daily.dev/og/posts/9FWV9MPUy.png og:image:alt: Behavioral testing for AI agents og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Behavioral testing for AI agents **[Red Hat Developer](https://daily.dev/sources/rhdev)** · 10 min read · 2 upvotes · 0 comments ## Summary Traditional unit tests can't catch behavioral failures in AI agents because they mock the LLM and test scaffolding, not model decisions. Behavioral testing fills this gap by sending real queries over HTTP to live agents and asserting observable outcomes: tool selection, response content, latency, and safety. The approach uses golden queries defined in YAML files, pass@k scoring to handle non-determinism, and per-agent thresholds. Safety is treated as a first-class behavioral concern, with adversarial payloads (prompt injection, PII leakage, policy violations) defined as data in YAML and parametrized into tests automatically. Behavioral tests run nightly against live agents in CI to catch regressions, while agent evaluation with LLM-as-judge handles reasoning quality separately. Practical guidance covers threshold calibration, golden query maintenance, and when behavioral testing is not the right tool. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### why do unit tests pass but my AI agent still calls the wrong tool in production Unit tests mock the LLM, so they test scaffolding like retry handling and response parsing rather than the model's actual decisions. An agent can return valid JSON and pass every unit test while still calling the wrong tool, hallucinating a nonexistent tool, or leaking PII, because tool selection is a decision made inside the model, not in the code being unit tested. _Teams debugging silent agent regressions can find testing approaches like this curated on daily.dev._ ### what is pass@k testing and why use it instead of pass@1 for AI agents Pass@k runs the same query k times (commonly k=8) and checks whether the success rate meets a per-agent threshold, such as 85%, rather than requiring every single run to succeed. This handles the non-deterministic execution paths of agents, where the same query can select different tools on different runs; for example, 7 out of 8 correct tool calls yields an 87.5% pass rate that clears an 85% threshold. _Anyone setting reliability thresholds for non-deterministic agents can track testing patterns like pass@k on daily.dev._ ### how do you test an AI agent for prompt injection resistance Send adversarial queries defined as data in a YAML file, covering direct injection, indirect injection, role-play jailbreaks, encoding tricks, and context manipulation, then scan the response for compliance signals like 'developer mode enabled' and check whether the agent hallucinated any tool calls it shouldn't have made. Adding a new attack only requires a one-line YAML addition, not new test code. _Developers hardening agents against prompt injection can follow security testing practices surfaced on daily.dev._ ## Similar posts on daily.dev - [Testing AI Agents: Validating Non-Deterministic Behavior](https://daily.dev/posts/testing-ai-agents-validating-non-deterministic-behavior-f2vc3iu26) · SitePoint · 1 upvotes · 0 comments - [Test Agents Using Agents](https://daily.dev/posts/test-agents-using-agents-lya7ynb5u) · Daily Dose of Data Science \| Avi Chawla \| Substack · 2 upvotes · 0 comments - [How We Are Testing Our Agents in Dev](https://daily.dev/posts/how-we-are-testing-our-agents-in-dev-bqwnixx0u) · Towards Data Science · 1 upvotes · 0 comments - [The 3 Layers of Testing AI Agents](https://daily.dev/posts/the-3-layers-of-testing-ai-agents-kf4cvo3yc) · The T-Shaped Dev · 4 upvotes · 0 comments - [How to automate the testing of AI agents](https://daily.dev/posts/how-to-automate-the-testing-of-ai-agents-dnu2r7umf) · InfoWorld · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#testing](https://daily.dev/tags/testing), [#ai-agents](https://daily.dev/tags/ai-agents), [#prompt-injection](https://daily.dev/tags/prompt-injection) [View this post on daily.dev](https://daily.dev/posts/behavioral-testing-for-ai-agents-9fwv9mpuy) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Behavioral testing for AI agents","url":"https://daily.dev/posts/behavioral-testing-for-ai-agents-9fwv9mpuy","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/behavioral-testing-for-ai-agents-9fwv9mpuy"},"datePublished":"2026-07-30T03:16:07.815Z","dateModified":"2026-09-14T08:34:19.606Z","description":"Traditional unit tests can't catch behavioral failures in AI agents because they mock the LLM and test scaffolding, not model decisions. Behavioral testing...","image":"https://media.daily.dev/image/upload/s--0_ODbtD2--/f_auto/v1722860399/public/Placeholder%2008","thumbnailUrl":"https://media.daily.dev/image/upload/s--0_ODbtD2--/f_auto/v1722860399/public/Placeholder%2008","isAccessibleForFree":true,"articleSection":"Red Hat Developer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Red Hat Developer","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/2a8895f3216c4900bdc455585a4ad3e4","url":"https://daily.dev/sources/rhdev"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/behavioral-testing-for-ai-agents-9fwv9mpuy","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,testing,ai-agents,prompt-injection","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Red Hat Developer","item":"https://daily.dev/sources/rhdev"},{"@type":"ListItem","position":3,"name":"Behavioral testing for AI agents"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/behavioral-testing-for-ai-agents-9fwv9mpuy#faq","mainEntity":[{"@type":"Question","name":"why do unit tests pass but my AI agent still calls the wrong tool in production","acceptedAnswer":{"@type":"Answer","text":"Unit tests mock the LLM, so they test scaffolding like retry handling and response parsing rather than the model's actual decisions. An agent can return valid JSON and pass every unit test while still calling the wrong tool, hallucinating a nonexistent tool, or leaking PII, because tool selection is a decision made inside the model, not in the code being unit tested. Teams debugging silent agent regressions can find testing approaches like this curated on daily.dev."}},{"@type":"Question","name":"what is pass@k testing and why use it instead of pass@1 for AI agents","acceptedAnswer":{"@type":"Answer","text":"Pass@k runs the same query k times (commonly k=8) and checks whether the success rate meets a per-agent threshold, such as 85%, rather than requiring every single run to succeed. This handles the non-deterministic execution paths of agents, where the same query can select different tools on different runs; for example, 7 out of 8 correct tool calls yields an 87.5% pass rate that clears an 85% threshold. Anyone setting reliability thresholds for non-deterministic agents can track testing patterns like pass@k on daily.dev."}},{"@type":"Question","name":"how do you test an AI agent for prompt injection resistance","acceptedAnswer":{"@type":"Answer","text":"Send adversarial queries defined as data in a YAML file, covering direct injection, indirect injection, role-play jailbreaks, encoding tricks, and context manipulation, then scan the response for compliance signals like 'developer mode enabled' and check whether the agent hallucinated any tool calls it shouldn't have made. Adding a new attack only requires a one-line YAML addition, not new test code. Developers hardening agents against prompt injection can follow security testing practices surfaced on daily.dev."}}]} ```