--- title: I Almost Built an AI QA Engineer. Then I Did the Math... description: A developer shares how they evaluated building an AI-powered QA engineer for their Playwright/TypeScript regression suite and chose a simpler, rule-based... canonical: https://daily.dev/posts/i-almost-built-an-ai-qa-engineer-then-i-did-the-math-and-built-the-boring-version-instead--6bkbbxthi twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: I Almost Built an AI QA Engineer. Then I Did the Math and Built the Boring Version Instead. | daily.dev og:description: A developer shares how they evaluated building an AI-powered QA engineer for their Playwright/TypeScript regression suite and chose a simpler, rule-based... og:url: https://daily.dev/posts/i-almost-built-an-ai-qa-engineer-then-i-did-the-math-and-built-the-boring-version-instead--6bkbbxthi og:image: https://api.daily.dev/og/posts/6BKBBxThi.png og:image:alt: I Almost Built an AI QA Engineer. Then I Did the Math and Built the Boring Version Instead. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # I Almost Built an AI QA Engineer. Then I Did the Math and Built the Boring Version Instead. **[Medium](https://daily.dev/sources/medium_js)** · 6 min read · 0 upvotes · 0 comments ## Summary A developer shares how they evaluated building an AI-powered QA engineer for their Playwright/TypeScript regression suite and chose a simpler, rule-based approach instead. Rather than implementing LLM-based failure triage, self-healing locators, and auto-bug-filing, they built two things: a regex-based failure classifier that categorizes test failures into actionable buckets, and a persistent run log that tracks flaky tests and failure trends over time. The post argues that in 2026, deliberately not reaching for AI is often the more sophisticated engineering choice — especially for small teams without the runway to absorb the costs of getting AI-powered tooling wrong. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Is it worth building a custom AI agent with DOM access and console log inspection for Playwright testing? Often not, because an official open-source MCP server with DOM access, console logs, and network inspection already exists, built by the Playwright team itself, and is available with a single npx command. Building a custom version duplicates work already solved for free by people who maintain Playwright full-time. _Weigh build-versus-reuse tradeoffs for AI testing tools like this with practical writeups on daily.dev._ ### How can I track flaky tests in a CI pipeline without using AI or machine learning? Append every test run to a log instead of overwriting it, then count failures across recent runs, for example flagging any test that failed in 3 of the last 5 runs as flaky. This same log can also surface which failure category occurs most often and in which part of the app, all through simple counting rather than a learning model. _Find pragmatic, low-cost testing patterns like this flaky-test tracker on daily.dev._ ## Similar posts on daily.dev - [The AI-augmented tester is here. So is a new problem: proving your tests actually work](https://daily.dev/posts/the-ai-augmented-tester-is-here-so-is-a-new-problem-proving-your-tests-actually-work-wnh2mhwcr) · SD Times · 0 upvotes · 1 comments - [When AI writes 100K lines of code, QA becomes the whole job](https://daily.dev/posts/when-ai-writes-100k-lines-of-code-qa-becomes-the-whole-job-oksv6lpnd) · The New Stack · 1 upvotes · 0 comments - [Your AI Testing Framework Might Be Passing Tests It Should Be Failing](https://daily.dev/posts/your-ai-testing-framework-might-be-passing-tests-it-should-be-failing-pdjtsnolg) · DevOps.com · 0 upvotes · 0 comments - [AI Isn’t Replacing Testers — It’s Replacing Outdated Testing](https://daily.dev/posts/ai-isn-t-replacing-testers-it-s-replacing-outdated-testing-gj6chv50o) · Medium · 0 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#testing](https://daily.dev/tags/testing), [#cicd](https://daily.dev/tags/cicd) [View this post on daily.dev](https://daily.dev/posts/i-almost-built-an-ai-qa-engineer-then-i-did-the-math-and-built-the-boring-version-instead--6bkbbxthi) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"I Almost Built an AI QA Engineer. Then I Did the Math and Built the Boring Version Instead.","url":"https://daily.dev/posts/i-almost-built-an-ai-qa-engineer-then-i-did-the-math-and-built-the-boring-version-instead--6bkbbxthi","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/i-almost-built-an-ai-qa-engineer-then-i-did-the-math-and-built-the-boring-version-instead--6bkbbxthi"},"datePublished":"2026-07-21T07:31:27.193Z","dateModified":"2026-09-13T18:40:07.956Z","description":"A developer shares how they evaluated building an AI-powered QA engineer for their Playwright/TypeScript regression suite and chose a simpler, rule-based...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5a8484b8483f49ddc1c680af70078422?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5a8484b8483f49ddc1c680af70078422?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/i-almost-built-an-ai-qa-engineer-then-i-did-the-math-and-built-the-boring-version-instead--6bkbbxthi","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,testing,cicd","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"I Almost Built an AI QA Engineer. Then I Did the Math and Built the Boring Version Instead."}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/i-almost-built-an-ai-qa-engineer-then-i-did-the-math-and-built-the-boring-version-instead--6bkbbxthi#faq","mainEntity":[{"@type":"Question","name":"Is it worth building a custom AI agent with DOM access and console log inspection for Playwright testing?","acceptedAnswer":{"@type":"Answer","text":"Often not, because an official open-source MCP server with DOM access, console logs, and network inspection already exists, built by the Playwright team itself, and is available with a single npx command. Building a custom version duplicates work already solved for free by people who maintain Playwright full-time. Weigh build-versus-reuse tradeoffs for AI testing tools like this with practical writeups on daily.dev."}},{"@type":"Question","name":"How can I track flaky tests in a CI pipeline without using AI or machine learning?","acceptedAnswer":{"@type":"Answer","text":"Append every test run to a log instead of overwriting it, then count failures across recent runs, for example flagging any test that failed in 3 of the last 5 runs as flaky. This same log can also surface which failure category occurs most often and in which part of the app, all through simple counting rather than a learning model. Find pragmatic, low-cost testing patterns like this flaky-test tracker on daily.dev."}}]} ```