--- title: Onboarding Agent from Scratch. Part 4: Prompts You Can... description: Part 4 of a series building an onboarding agent from scratch focuses on treating prompts as versioned artifacts with regression test suites rather than... canonical: https://daily.dev/posts/onboarding-agent-from-scratch-part-4-prompts-you-can-test-not-just-tweak-t5upqegtx twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Onboarding Agent from Scratch. Part 4: Prompts You Can Test, Not Just Tweak | daily.dev og:description: Part 4 of a series building an onboarding agent from scratch focuses on treating prompts as versioned artifacts with regression test suites rather than... og:url: https://daily.dev/posts/onboarding-agent-from-scratch-part-4-prompts-you-can-test-not-just-tweak-t5upqegtx og:image: https://api.daily.dev/og/posts/t5uPqegTX.png og:image:alt: Onboarding Agent from Scratch. Part 4: Prompts You Can Test, Not Just Tweak og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Onboarding Agent from Scratch. Part 4: Prompts You Can Test, Not Just Tweak **[Medium](https://daily.dev/sources/medium_js)** · 5 min read · 0 upvotes · 0 comments ## Summary Part 4 of a series building an onboarding agent from scratch focuses on treating prompts as versioned artifacts with regression test suites rather than editable strings. Two prompt versions (V1 and V2) are stored as named constants in a dedicated module, and a pytest suite runs both against three question types — in-scope, out-of-scope, and partial-coverage — checking structural response properties (confidence level, source presence) rather than exact wording. The key distinction drawn is between regression tests (binary pass/fail for catching drift) and evals (graded quality assessments). The live run passed all six cases, revealing that claude-opus-4 already handled partial-coverage grounding well without explicit instruction, illustrating that a green regression suite proves stability but not that a prompt change had any effect. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What's the difference between a regression test and an eval when testing LLM prompts? A regression test is binary, checking whether something that used to pass starts failing, and is cheap to write and run, making it good for catching obvious breakage. An eval is a graded quality assessment, often scored 1-5 and sometimes judged by another model, answering how good an output is rather than just whether it broke. Evals cost more to build and suit quality tuning, while regression suites suit catching drift. _daily.dev surfaces practical breakdowns like this for developers deciding how to test their own LLM prompts._ ### How can I test that an LLM gives lower confidence when context only partially answers a question? Write a regression test with a genuinely mixed question that combines a covered topic with an uncovered one, then assert the response's confidence field is medium or low, never high, while still including a non-empty source. In one implementation, combining remote working hours (covered by an FAQ) with a home-office stipend (not mentioned) reliably triggered medium or low confidence across two prompt versions. _developers building grounded Q&A agents can track approaches like this on daily.dev while shaping their own test cases._ ### Why might adding an explicit instruction to a prompt not change a model's structured output in testing? An added instruction can shift emphasis in a free-text answer field without changing the structural fields a regression suite actually checks, such as confidence or source. In one test, an instruction telling a model to name missing context and reserve high confidence for full coverage produced no change in confidence or source, because the baseline model, claude-opus-4-8, already handled partial-coverage grounding well unprompted; only an eval scoring free-text phrasing would have caught the difference. _daily.dev helps developers compare prompt-tuning results like this when deciding whether a change actually mattered._ ## Similar posts on daily.dev - [Prompts as Code: Versioning & Testing Guide \(2026\)](https://daily.dev/posts/prompts-as-code-versioning-testing-guide-2026--tjwfufkgq) · Alex CloudStar · 0 upvotes · 0 comments - [Prompt Engineering Fails Quietly — Prompt Regression Is Why](https://daily.dev/posts/prompt-engineering-fails-quietly-prompt-regression-is-why-niy5iiaz2) · Towards Data Science · 2 upvotes · 1 comments - [Prompt Engineering for AI Agents: From Demo to Production](https://daily.dev/posts/prompt-engineering-for-ai-agents-from-demo-to-production-vfcpohk9t) · Medium · 1 upvotes · 0 comments - [Testing AI prompts and comparing models with promptfoo](https://daily.dev/posts/testing-ai-prompts-and-comparing-models-with-promptfoo-mxhomphpc) · Tim Deschryver · 1 upvotes · 1 comments --- Tags: [#python](https://daily.dev/tags/python), [#testing](https://daily.dev/tags/testing), [#ai-agents](https://daily.dev/tags/ai-agents), [#claude](https://daily.dev/tags/claude), [#prompt-engineering](https://daily.dev/tags/prompt-engineering) [View this post on daily.dev](https://daily.dev/posts/onboarding-agent-from-scratch-part-4-prompts-you-can-test-not-just-tweak-t5upqegtx) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Onboarding Agent from Scratch. Part 4: Prompts You Can Test, Not Just Tweak","url":"https://daily.dev/posts/onboarding-agent-from-scratch-part-4-prompts-you-can-test-not-just-tweak-t5upqegtx","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/onboarding-agent-from-scratch-part-4-prompts-you-can-test-not-just-tweak-t5upqegtx"},"datePublished":"2026-07-22T19:07:40.904Z","dateModified":"2026-09-14T07:38:41.851Z","description":"Part 4 of a series building an onboarding agent from scratch focuses on treating prompts as versioned artifacts with regression test suites rather than...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/61e32be7f6d2f9e0ba75302e1133986d?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/61e32be7f6d2f9e0ba75302e1133986d?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/onboarding-agent-from-scratch-part-4-prompts-you-can-test-not-just-tweak-t5upqegtx","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,testing,ai-agents,claude,prompt-engineering","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"Onboarding Agent from Scratch. Part 4: Prompts You Can Test, Not Just Tweak"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/onboarding-agent-from-scratch-part-4-prompts-you-can-test-not-just-tweak-t5upqegtx#faq","mainEntity":[{"@type":"Question","name":"What's the difference between a regression test and an eval when testing LLM prompts?","acceptedAnswer":{"@type":"Answer","text":"A regression test is binary, checking whether something that used to pass starts failing, and is cheap to write and run, making it good for catching obvious breakage. An eval is a graded quality assessment, often scored 1-5 and sometimes judged by another model, answering how good an output is rather than just whether it broke. Evals cost more to build and suit quality tuning, while regression suites suit catching drift. daily.dev surfaces practical breakdowns like this for developers deciding how to test their own LLM prompts."}},{"@type":"Question","name":"How can I test that an LLM gives lower confidence when context only partially answers a question?","acceptedAnswer":{"@type":"Answer","text":"Write a regression test with a genuinely mixed question that combines a covered topic with an uncovered one, then assert the response's confidence field is medium or low, never high, while still including a non-empty source. In one implementation, combining remote working hours (covered by an FAQ) with a home-office stipend (not mentioned) reliably triggered medium or low confidence across two prompt versions. developers building grounded Q&A agents can track approaches like this on daily.dev while shaping their own test cases."}},{"@type":"Question","name":"Why might adding an explicit instruction to a prompt not change a model's structured output in testing?","acceptedAnswer":{"@type":"Answer","text":"An added instruction can shift emphasis in a free-text answer field without changing the structural fields a regression suite actually checks, such as confidence or source. In one test, an instruction telling a model to name missing context and reserve high confidence for full coverage produced no change in confidence or source, because the baseline model, claude-opus-4-8, already handled partial-coverage grounding well unprompted; only an eval scoring free-text phrasing would have caught the difference. daily.dev helps developers compare prompt-tuning results like this when deciding whether a change actually mattered."}}]} ```