--- title: How Do You Set Up a Regression Gate for Prompt Changes... description: A practical walkthrough for building a regression gate that evaluates LLM prompt or model changes in CI, treating quality drops the way a test suite treats... canonical: https://daily.dev/posts/how-do-you-set-up-a-regression-gate-for-prompt-changes-in-ci--qva1vazg3 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How Do You Set Up a Regression Gate for Prompt Changes in CI? | daily.dev og:description: A practical walkthrough for building a regression gate that evaluates LLM prompt or model changes in CI, treating quality drops the way a test suite treats... og:url: https://daily.dev/posts/how-do-you-set-up-a-regression-gate-for-prompt-changes-in-ci--qva1vazg3 og:image: https://api.daily.dev/og/posts/qvA1vaZg3.png og:image:alt: How Do You Set Up a Regression Gate for Prompt Changes in CI? og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How Do You Set Up a Regression Gate for Prompt Changes in CI? **[HEARTBEAT](https://daily.dev/sources/hrb)** · 8 min read · 0 upvotes · 0 comments ## Summary A practical walkthrough for building a regression gate that evaluates LLM prompt or model changes in CI, treating quality drops the way a test suite treats failures. Covers fixing a versioned evaluation dataset, wrapping the Opik Python SDK's evaluate function in a CI script, reading metric direction correctly (AnswerRelevance higher-is-better vs Hallucination higher-is-worse), setting absolute and relative thresholds before seeing a failure, and keeping the gate cheap via small datasets, smaller judge models, and tuned concurrency. Includes concrete code for dataset creation, evaluation, and baseline comparison using Opik SDK version 2.2.47. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I set up a regression gate in CI that fails the build when an LLM prompt change degrades quality? Score a fixed, versioned dataset on every prompt or model change using Opik's evaluate function, then compare the aggregate metric score against a baseline stored in version control. Run both an absolute check (score below a floor like 0.75) and a relative check (drop of more than a set amount, e.g. 0.05, from baseline). Exit the script with a non-zero status to fail the CI step. _Teams shipping LLM features can track CI patterns like this for catching prompt regressions on daily.dev._ ### Why does the Hallucination metric need a different threshold direction than AnswerRelevance in an LLM eval gate? AnswerRelevance returns 0.0 to 1.0 where higher is better, so it needs a floor and fails when the mean drops below it. Hallucination returns 1.0 when a hallucination is detected and 0.0 when it is not, so higher is worse and it needs a ceiling, failing when the mean rises above the limit. Treating both metrics the same way produces a gate that passes even as hallucinations increase. _Developers wiring judge metrics into pipelines can compare directional gotchas like this via daily.dev._ ### How many items and which judge model should a per-commit LLM regression gate use to stay fast and cheap? Cap the per-commit gate dataset at 50 to 100 items, which is enough to detect a real regression, and reserve larger evaluation suites for a scheduled nightly run. Use a smaller judge model such as gpt-4o-mini for lower cost and speed, reserving larger judges for the nightly suite. A well-tuned gate on 50 items with a small judge model completes in under 3 minutes and costs well under $1 per run. _Anyone budgeting CI evaluation costs for LLM pipelines can find sizing guidance like this on daily.dev._ ## Similar posts on daily.dev - [Why traditional CI/CD fails for LLMs \(and the release gates we built to fix it\)](https://daily.dev/posts/why-traditional-ci-cd-fails-for-llms-and-the-release-gates-we-built-to-fix-it--e3jggw3ud) · The New Stack · 0 upvotes · 0 comments - [How Do You Build an LLM Evaluation Dataset from Production Traces?](https://daily.dev/posts/how-do-you-build-an-llm-evaluation-dataset-from-production-traces--pmibina7y) · HEARTBEAT · 2 upvotes · 0 comments - [Integrating LLM Evaluations into Your CI Pipeline: Best Practices for AI Teams](https://daily.dev/posts/integrating-llm-evaluations-into-your-ci-pipeline-best-practices-for-ai-teams-kg95szbwx) · PromptLayer Blog · 0 upvotes · 0 comments - [Onboarding Agent from Scratch. Part 4: Prompts You Can Test, Not Just Tweak](https://daily.dev/posts/onboarding-agent-from-scratch-part-4-prompts-you-can-test-not-just-tweak-t5upqegtx) · Medium · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#testing](https://daily.dev/tags/testing), [#cicd](https://daily.dev/tags/cicd), [#prompt-engineering](https://daily.dev/tags/prompt-engineering) [View this post on daily.dev](https://daily.dev/posts/how-do-you-set-up-a-regression-gate-for-prompt-changes-in-ci--qva1vazg3) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How Do You Set Up a Regression Gate for Prompt Changes in CI?","url":"https://daily.dev/posts/how-do-you-set-up-a-regression-gate-for-prompt-changes-in-ci--qva1vazg3","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-do-you-set-up-a-regression-gate-for-prompt-changes-in-ci--qva1vazg3"},"datePublished":"2026-09-01T22:53:18.215Z","dateModified":"2026-09-14T06:20:55.583Z","description":"A practical walkthrough for building a regression gate that evaluates LLM prompt or model changes in CI, treating quality drops the way a test suite treats...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/aad2aeed548bd8708260f1a76f989369?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/aad2aeed548bd8708260f1a76f989369?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"HEARTBEAT","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"HEARTBEAT","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hrb","url":"https://daily.dev/sources/hrb"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-do-you-set-up-a-regression-gate-for-prompt-changes-in-ci--qva1vazg3","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,testing,cicd,prompt-engineering","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"HEARTBEAT","item":"https://daily.dev/sources/hrb"},{"@type":"ListItem","position":3,"name":"How Do You Set Up a Regression Gate for Prompt Changes in CI?"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-do-you-set-up-a-regression-gate-for-prompt-changes-in-ci--qva1vazg3#faq","mainEntity":[{"@type":"Question","name":"How do I set up a regression gate in CI that fails the build when an LLM prompt change degrades quality?","acceptedAnswer":{"@type":"Answer","text":"Score a fixed, versioned dataset on every prompt or model change using Opik's evaluate function, then compare the aggregate metric score against a baseline stored in version control. Run both an absolute check (score below a floor like 0.75) and a relative check (drop of more than a set amount, e.g. 0.05, from baseline). Exit the script with a non-zero status to fail the CI step. Teams shipping LLM features can track CI patterns like this for catching prompt regressions on daily.dev."}},{"@type":"Question","name":"Why does the Hallucination metric need a different threshold direction than AnswerRelevance in an LLM eval gate?","acceptedAnswer":{"@type":"Answer","text":"AnswerRelevance returns 0.0 to 1.0 where higher is better, so it needs a floor and fails when the mean drops below it. Hallucination returns 1.0 when a hallucination is detected and 0.0 when it is not, so higher is worse and it needs a ceiling, failing when the mean rises above the limit. Treating both metrics the same way produces a gate that passes even as hallucinations increase. Developers wiring judge metrics into pipelines can compare directional gotchas like this via daily.dev."}},{"@type":"Question","name":"How many items and which judge model should a per-commit LLM regression gate use to stay fast and cheap?","acceptedAnswer":{"@type":"Answer","text":"Cap the per-commit gate dataset at 50 to 100 items, which is enough to detect a real regression, and reserve larger evaluation suites for a scheduled nightly run. Use a smaller judge model such as gpt-4o-mini for lower cost and speed, reserving larger judges for the nightly suite. A well-tuned gate on 50 items with a small judge model completes in under 3 minutes and costs well under $1 per run. Anyone budgeting CI evaluation costs for LLM pipelines can find sizing guidance like this on daily.dev."}}]} ```