--- title: I Built PromptAB Because “This Prompt Looks Better”... description: PromptAB is a Python library for A/B testing LLM system prompts instead of guessing which one 'looks better'. You give it two prompts, a query, and optional... canonical: https://daily.dev/posts/i-built-promptab-because-this-prompt-looks-better-isn-t-a-metric-t55fwuxuw twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: I Built PromptAB Because “This Prompt Looks Better” Isn’t a Metric | daily.dev og:description: PromptAB is a Python library for A/B testing LLM system prompts instead of guessing which one 'looks better'. You give it two prompts, a query, and optional... og:url: https://daily.dev/posts/i-built-promptab-because-this-prompt-looks-better-isn-t-a-metric-t55fwuxuw og:image: https://api.daily.dev/og/posts/T55fWuXuw.png og:image:alt: I Built PromptAB Because “This Prompt Looks Better” Isn’t a Metric og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # I Built PromptAB Because “This Prompt Looks Better” Isn’t a Metric **[Medium](https://daily.dev/sources/medium_js)** · 3 min read · 0 upvotes · 0 comments ## Summary PromptAB is a Python library for A/B testing LLM system prompts instead of guessing which one 'looks better'. You give it two prompts, a query, and optional criteria, and it compares quality, criteria adherence, latency, cost, and consistency across runs, then outputs HTML or JSON reports and suggests an improved third prompt. While testing it, the author's LLM judge scored a factually wrong answer (Kapil Dev scoring 39 runs) at 100/100, and the correct answer (15 runs) at only 98.5/100, showing that LLM-based evaluation scores don't guarantee factual accuracy. The library is available via GitHub and PyPI. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I A/B test two LLM system prompts in Python instead of just eyeballing the outputs? A Python library called PromptAB runs two system prompts against a query and compares them on quality, criteria adherence, latency, cost, and consistency across multiple runs. It generates HTML or JSON reports and suggests an improved third prompt. Usage is a single `compare()` call passing prompt_a, prompt_b, a query, and optional evaluation criteria. _daily.dev surfaces tools like this for developers formalizing their prompt testing workflow._ ### Can I trust an LLM-as-judge score to tell me which response is factually correct? No, an LLM judge scoring responses can rate a factually wrong answer higher than a correct one. In one test, a judge gave an answer claiming Kapil Dev scored 39 runs in the 1983 World Cup final a 100/100, while the correct answer of 15 runs scored only 98.5/100. Verified reference answers, clear criteria, and human review still matter alongside automated scoring. _developers relying on AI evaluation pipelines can track these pitfalls through daily.dev._ ## Similar posts on daily.dev - [Automate Writing Your LLM Prompts](https://daily.dev/posts/automate-writing-your-llm-prompts-yv2mxht51) · Towards Data Science · 0 upvotes · 0 comments - [Testing AI prompts and comparing models with promptfoo](https://daily.dev/posts/testing-ai-prompts-and-comparing-models-with-promptfoo-mxhomphpc) · Tim Deschryver · 1 upvotes · 1 comments - [Track, compare, and optimize your LLM prompts with Datadog LLM Observability](https://daily.dev/posts/track-compare-and-optimize-your-llm-prompts-with-datadog-llm-observability-gluqvoqay) · Datadog · 1 upvotes · 0 comments - [Prompts as Code: Versioning & Testing Guide \(2026\)](https://daily.dev/posts/prompts-as-code-versioning-testing-guide-2026--tjwfufkgq) · Alex CloudStar · 0 upvotes · 0 comments - [The PromptLayer Way: Building LLM Applications Through Reflective Iteration](https://daily.dev/posts/the-promptlayer-way-building-llm-applications-through-reflective-iteration-ed6ebsnrd) · PromptLayer Blog · 0 upvotes · 0 comments --- Tags: [#python](https://daily.dev/tags/python), [#open-source](https://daily.dev/tags/open-source), [#llm](https://daily.dev/tags/llm), [#testing](https://daily.dev/tags/testing), [#prompt-engineering](https://daily.dev/tags/prompt-engineering) [View this post on daily.dev](https://daily.dev/posts/i-built-promptab-because-this-prompt-looks-better-isn-t-a-metric-t55fwuxuw) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"I Built PromptAB Because “This Prompt Looks Better” Isn’t a Metric","url":"https://daily.dev/posts/i-built-promptab-because-this-prompt-looks-better-isn-t-a-metric-t55fwuxuw","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/i-built-promptab-because-this-prompt-looks-better-isn-t-a-metric-t55fwuxuw"},"datePublished":"2026-11-01T02:12:23.644Z","dateModified":"2026-11-01T02:14:10.216Z","description":"PromptAB is a Python library for A/B testing LLM system prompts instead of guessing which one 'looks better'. You give it two prompts, a query, and optional...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d6ab0a9a223ee01bc82d647f54feb2d2?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d6ab0a9a223ee01bc82d647f54feb2d2?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/i-built-promptab-because-this-prompt-looks-better-isn-t-a-metric-t55fwuxuw","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,open-source,llm,testing,prompt-engineering","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"I Built PromptAB Because “This Prompt Looks Better” Isn’t a Metric"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/i-built-promptab-because-this-prompt-looks-better-isn-t-a-metric-t55fwuxuw#faq","mainEntity":[{"@type":"Question","name":"How can I A/B test two LLM system prompts in Python instead of just eyeballing the outputs?","acceptedAnswer":{"@type":"Answer","text":"A Python library called PromptAB runs two system prompts against a query and compares them on quality, criteria adherence, latency, cost, and consistency across multiple runs. It generates HTML or JSON reports and suggests an improved third prompt. Usage is a single `compare()` call passing prompt_a, prompt_b, a query, and optional evaluation criteria. daily.dev surfaces tools like this for developers formalizing their prompt testing workflow."}},{"@type":"Question","name":"Can I trust an LLM-as-judge score to tell me which response is factually correct?","acceptedAnswer":{"@type":"Answer","text":"No, an LLM judge scoring responses can rate a factually wrong answer higher than a correct one. In one test, a judge gave an answer claiming Kapil Dev scored 39 runs in the 1983 World Cup final a 100/100, while the correct answer of 15 runs scored only 98.5/100. Verified reference answers, clear criteria, and human review still matter alongside automated scoring. developers relying on AI evaluation pipelines can track these pitfalls through daily.dev."}}]} ```