--- title: AI Testing Is Bigger Than You Think, 5 Areas Testers... description: A principal software quality engineer discusses a five-area framework for AI testing: using AI to augment testing, evaluating AI output for trust, governing AI... canonical: https://daily.dev/posts/ai-testing-is-bigger-than-you-think-5-areas-testers-must-own-v8tj8xtcs twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: AI Testing Is Bigger Than You Think, 5 Areas Testers Must Own | daily.dev og:description: A principal software quality engineer discusses a five-area framework for AI testing: using AI to augment testing, evaluating AI output for trust, governing AI... og:url: https://daily.dev/posts/ai-testing-is-bigger-than-you-think-5-areas-testers-must-own-v8tj8xtcs og:image: https://api.daily.dev/og/posts/V8Tj8xtcs.png og:image:alt: AI Testing Is Bigger Than You Think, 5 Areas Testers Must Own og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # AI Testing Is Bigger Than You Think, 5 Areas Testers Must Own **[Automation Testing with Joe Colantonio](https://daily.dev/sources/joecolantonio)** · 40 min read · 0 upvotes · 0 comments ## Summary A principal software quality engineer discusses a five-area framework for AI testing: using AI to augment testing, evaluating AI output for trust, governing AI development quality, testing products with embedded AI features, and testing AI models themselves. Key themes include AI's tendency to be sycophantic (agreeing with test strategies rather than critiquing them), the risk of infinite loops when engaging with agentic AI, breaking fully-autonomous workflows into smaller delegated steps to preserve reliability, and the importance of testers building strong evaluation and business-domain skills alongside AI literacy. Concrete examples include a broken GitHub Copilot MCP tool for test case generation and a RAG-based watering guide chatbot that hallucinated incorrect answers despite having accurate source material. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does AI always say my test strategy looks good even when it might not be? Large language models tend to behave like people-pleasers, agreeing with input rather than critically evaluating it. Pasting a test strategy into an AI assistant will produce an affirming response like 'this looks great' nearly every time, which provides false confidence rather than genuine validation, especially since readers tend to focus on the agreeable opening lines of verbose AI responses. _Testers weighing how much to trust AI feedback can follow ongoing discussion on AI testing practices via daily.dev._ ### Why does breaking an AI automation task into smaller steps improve reliability compared to a fully autonomous pipeline? Splitting a task, such as generating test cases and then uploading them to a tracker, into separate delegated steps trades some speed for accuracy because it forces a human review checkpoint between generation and execution. A fully autonomous pipeline that both generates and uploads content risks compounding errors, like incomplete test cataloging, that a mid-process review would catch. _Teams designing AI-assisted testing pipelines can track patterns like this through daily.dev._ ### Why did a RAG-based chatbot give an incorrect answer even when it was restricted to a single accurate source document? A retrieval-augmented generation model trained only on a two-page watering guide PDF still answered 'water tomatoes when the soil feels moist,' which is backwards advice, despite being scoped to that single source. Diagnosing the error required using an observability framework (LangChain) to trace how the input was processed and where the retrieval or generation step went wrong. _Developers debugging unexpected RAG outputs can follow observability techniques for LLM pipelines on daily.dev._ ## Similar posts on daily.dev - [Testing AI systems: a practical guide for engineering teams](https://daily.dev/posts/testing-ai-systems-a-practical-guide-for-engineering-teams-95owulvhm) · Netguru · 0 upvotes · 0 comments - [Testers, testing and the future: A Bifurcation into Testing AI and AI-powered Testing.](https://daily.dev/posts/testers-testing-and-the-future-a-bifurcation-into-testing-ai-and-ai-powered-testing--xgg52y1q9) · Scott Logic · 0 upvotes · 0 comments - [The Future of Software Testing in an AI-Driven World](https://daily.dev/posts/the-future-of-software-testing-in-an-ai-driven-world-r9ltatite) · Software Testing Magazine · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#testing](https://daily.dev/tags/testing), [#github](https://daily.dev/tags/github) [View this post on daily.dev](https://daily.dev/posts/ai-testing-is-bigger-than-you-think-5-areas-testers-must-own-v8tj8xtcs) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"AI Testing Is Bigger Than You Think, 5 Areas Testers Must Own","url":"https://daily.dev/posts/ai-testing-is-bigger-than-you-think-5-areas-testers-must-own-v8tj8xtcs","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/ai-testing-is-bigger-than-you-think-5-areas-testers-must-own-v8tj8xtcs"},"datePublished":"2026-09-01T15:52:56.966Z","dateModified":"2026-09-01T15:58:27.338Z","description":"A principal software quality engineer discusses a five-area framework for AI testing: using AI to augment testing, evaluating AI output for trust, governing AI...","image":"https://i.ytimg.com/vi/7W2a42MEy3M/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/7W2a42MEy3M/sddefault.jpg","isAccessibleForFree":true,"articleSection":"Automation Testing with Joe Colantonio","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Automation Testing with Joe Colantonio","logo":"https://media.daily.dev/image/upload/s--euSb6E7Q--/f_auto,q_auto/v1780213680/logos/joecolantonio?_a=BAMAMiWQ0","url":"https://daily.dev/sources/joecolantonio"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/ai-testing-is-bigger-than-you-think-5-areas-testers-must-own-v8tj8xtcs","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,testing,github","timeRequired":"PT40M","video":{"@type":"VideoObject","name":"AI Testing Is Bigger Than You Think, 5 Areas Testers Must Own","description":"A principal software quality engineer discusses a five-area framework for AI testing: using AI to augment testing, evaluating AI output for trust, governing AI...","thumbnailUrl":"https://i.ytimg.com/vi/7W2a42MEy3M/sddefault.jpg","uploadDate":"2026-09-01T15:52:56.966Z","duration":"PT40M","url":"https://api.daily.dev/r/V8Tj8xtcs","embedUrl":"https://www.youtube.com/embed/7W2a42MEy3M"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Automation Testing with Joe Colantonio","item":"https://daily.dev/sources/joecolantonio"},{"@type":"ListItem","position":3,"name":"AI Testing Is Bigger Than You Think, 5 Areas Testers Must Own"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/ai-testing-is-bigger-than-you-think-5-areas-testers-must-own-v8tj8xtcs#faq","mainEntity":[{"@type":"Question","name":"Why does AI always say my test strategy looks good even when it might not be?","acceptedAnswer":{"@type":"Answer","text":"Large language models tend to behave like people-pleasers, agreeing with input rather than critically evaluating it. Pasting a test strategy into an AI assistant will produce an affirming response like 'this looks great' nearly every time, which provides false confidence rather than genuine validation, especially since readers tend to focus on the agreeable opening lines of verbose AI responses. Testers weighing how much to trust AI feedback can follow ongoing discussion on AI testing practices via daily.dev."}},{"@type":"Question","name":"Why does breaking an AI automation task into smaller steps improve reliability compared to a fully autonomous pipeline?","acceptedAnswer":{"@type":"Answer","text":"Splitting a task, such as generating test cases and then uploading them to a tracker, into separate delegated steps trades some speed for accuracy because it forces a human review checkpoint between generation and execution. A fully autonomous pipeline that both generates and uploads content risks compounding errors, like incomplete test cataloging, that a mid-process review would catch. Teams designing AI-assisted testing pipelines can track patterns like this through daily.dev."}},{"@type":"Question","name":"Why did a RAG-based chatbot give an incorrect answer even when it was restricted to a single accurate source document?","acceptedAnswer":{"@type":"Answer","text":"A retrieval-augmented generation model trained only on a two-page watering guide PDF still answered 'water tomatoes when the soil feels moist,' which is backwards advice, despite being scoped to that single source. Diagnosing the error required using an observability framework (LangChain) to trace how the input was processed and where the retrieval or generation step went wrong. Developers debugging unexpected RAG outputs can follow observability techniques for LLM pipelines on daily.dev."}}]} ```