--- title: How to Build a Self-Evaluating AI System: Automated... description: A comprehensive tutorial builds a three-layer evaluation pipeline for LLM applications in Python: fast deterministic checks (JSON validity, length,... canonical: https://daily.dev/posts/how-to-build-a-self-evaluating-ai-system-automated-testing-and-evaluation-pipelines-for-llm-applica-yppnqlvra twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How to Build a Self-Evaluating AI System: Automated Testing and Evaluation Pipelines for LLM Applications | daily.dev og:description: A comprehensive tutorial builds a three-layer evaluation pipeline for LLM applications in Python: fast deterministic checks (JSON validity, length,... og:url: https://daily.dev/posts/how-to-build-a-self-evaluating-ai-system-automated-testing-and-evaluation-pipelines-for-llm-applica-yppnqlvra og:image: https://api.daily.dev/og/posts/yPpnqlVrA.png og:image:alt: How to Build a Self-Evaluating AI System: Automated Testing and Evaluation Pipelines for LLM Applications og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How to Build a Self-Evaluating AI System: Automated Testing and Evaluation Pipelines for LLM Applications **[freeCodeCamp](https://daily.dev/sources/freecodecamp)** · 29 min read · 0 upvotes · 0 comments ## Summary A comprehensive tutorial builds a three-layer evaluation pipeline for LLM applications in Python: fast deterministic checks (JSON validity, length, hallucinated links, refusals), LLM-as-judge scoring using gpt-4o-mini against concrete rubrics with multi-judge consensus for reliability, and periodic human evaluation with Cohen's Kappa to measure annotator agreement. It also covers building golden datasets for regression testing, wiring evaluations into CI/CD to block regressions, and using a paired t-test to confirm that prompt or model changes produce statistically significant improvements rather than noise. Full working code is provided for each component, culminating in an orchestrator class that chains all three layers together. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I check if two people scoring the same LLM output actually agree with each other? Use Cohen's Kappa, which measures inter-annotator agreement adjusted for chance agreement, via scikit-learn's cohen_kappa_score function. A kappa above 0.6 indicates substantial agreement; below that, the scoring rubric itself is likely ambiguous rather than the annotators being inconsistent. Refining the rubric with more concrete examples per score level typically takes two to three iteration rounds to fix. _Teams calibrating LLM scoring rubrics can track evaluation techniques like this one on daily.dev._ ### How do I know if a prompt change actually improved my LLM app's eval scores or if it's just noise? Run a paired t-test comparing scores on the same golden dataset questions before and after the change, using scipy's stats.ttest_rel. If the resulting p-value is below 0.05, the improvement is statistically significant and safe to deploy; with only 50-100 examples, a score jump like 3.8 to 4.0 can easily be random variance rather than a real gain. _Developers validating prompt or model changes can follow statistical testing approaches like this on daily.dev._ ### Why does asking an LLM to rate a response from 1 to 10 give inconsistent scores? Because the model has no shared definition of what each number means, so a 7 on one run can become a 5 on the next. Replacing an open numeric scale with a rubric that gives concrete, observable anchor descriptions for each score level (for example, distinguishing 'completely off-topic' from a vague 'kind of bad') makes an LLM judge's scoring far more consistent across runs. _Anyone designing LLM-as-judge scoring can compare rubric approaches like this via daily.dev._ ## Similar posts on daily.dev - [AI Evaluation Engineering: Build a Production-Grade LLM Evaluation Platform from Scratch \[Full Handbook\]](https://daily.dev/posts/ai-evaluation-engineering-build-a-production-grade-llm-evaluation-platform-from-scratch-full-handb-ju9v0rogq) · freeCodeCamp · 5 upvotes · 0 comments - [Notes on LLM Evaluation](https://daily.dev/posts/notes-on-llm-evaluation-xbcfojzcc) · Towards Data Science · 1 upvotes · 0 comments --- Tags: [#python](https://daily.dev/tags/python), [#llm](https://daily.dev/tags/llm), [#testing](https://daily.dev/tags/testing), [#cicd](https://daily.dev/tags/cicd) [View this post on daily.dev](https://daily.dev/posts/how-to-build-a-self-evaluating-ai-system-automated-testing-and-evaluation-pipelines-for-llm-applica-yppnqlvra) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How to Build a Self-Evaluating AI System: Automated Testing and Evaluation Pipelines for LLM Applications","url":"https://daily.dev/posts/how-to-build-a-self-evaluating-ai-system-automated-testing-and-evaluation-pipelines-for-llm-applica-yppnqlvra","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-to-build-a-self-evaluating-ai-system-automated-testing-and-evaluation-pipelines-for-llm-applica-yppnqlvra"},"datePublished":"2026-09-11T18:23:05.907Z","dateModified":"2026-09-11T18:23:34.080Z","description":"A comprehensive tutorial builds a three-layer evaluation pipeline for LLM applications in Python: fast deterministic checks (JSON validity, length,...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1f568980a1b7d0c43eb4b9c5e52b0b1d?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1f568980a1b7d0c43eb4b9c5e52b0b1d?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"freeCodeCamp","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"freeCodeCamp","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1628412854/logos/freecodecamp","url":"https://daily.dev/sources/freecodecamp"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-to-build-a-self-evaluating-ai-system-automated-testing-and-evaluation-pipelines-for-llm-applica-yppnqlvra","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,llm,testing,cicd","timeRequired":"PT29M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"freeCodeCamp","item":"https://daily.dev/sources/freecodecamp"},{"@type":"ListItem","position":3,"name":"How to Build a Self-Evaluating AI System: Automated Testing and Evaluation Pipelines for LLM Applications"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-to-build-a-self-evaluating-ai-system-automated-testing-and-evaluation-pipelines-for-llm-applica-yppnqlvra#faq","mainEntity":[{"@type":"Question","name":"How do I check if two people scoring the same LLM output actually agree with each other?","acceptedAnswer":{"@type":"Answer","text":"Use Cohen's Kappa, which measures inter-annotator agreement adjusted for chance agreement, via scikit-learn's cohen_kappa_score function. A kappa above 0.6 indicates substantial agreement; below that, the scoring rubric itself is likely ambiguous rather than the annotators being inconsistent. Refining the rubric with more concrete examples per score level typically takes two to three iteration rounds to fix. Teams calibrating LLM scoring rubrics can track evaluation techniques like this one on daily.dev."}},{"@type":"Question","name":"How do I know if a prompt change actually improved my LLM app's eval scores or if it's just noise?","acceptedAnswer":{"@type":"Answer","text":"Run a paired t-test comparing scores on the same golden dataset questions before and after the change, using scipy's stats.ttest_rel. If the resulting p-value is below 0.05, the improvement is statistically significant and safe to deploy; with only 50-100 examples, a score jump like 3.8 to 4.0 can easily be random variance rather than a real gain. Developers validating prompt or model changes can follow statistical testing approaches like this on daily.dev."}},{"@type":"Question","name":"Why does asking an LLM to rate a response from 1 to 10 give inconsistent scores?","acceptedAnswer":{"@type":"Answer","text":"Because the model has no shared definition of what each number means, so a 7 on one run can become a 5 on the next. Replacing an open numeric scale with a rubric that gives concrete, observable anchor descriptions for each score level (for example, distinguishing 'completely off-topic' from a vague 'kind of bad') makes an LLM judge's scoring far more consistent across runs. Anyone designing LLM-as-judge scoring can compare rubric approaches like this via daily.dev."}}]} ```