--- title: How to Evaluate AI Agents with an LLM-as-a-Judge Harness... description: A step-by-step guide to building a repeatable evaluation harness for local AI agents using Python, LangChain v1, Ollama, and Qwen. The harness runs test cases... canonical: https://daily.dev/posts/how-to-evaluate-ai-agents-with-an-llm-as-a-judge-harness-in-python-frgzj1tat twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python | daily.dev og:description: A step-by-step guide to building a repeatable evaluation harness for local AI agents using Python, LangChain v1, Ollama, and Qwen. The harness runs test cases... og:url: https://daily.dev/posts/how-to-evaluate-ai-agents-with-an-llm-as-a-judge-harness-in-python-frgzj1tat og:image: https://api.daily.dev/og/posts/FRGZJ1TAT.png og:image:alt: How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python **[freeCodeCamp](https://daily.dev/sources/freecodecamp)** · 11 min read · 3 upvotes · 0 comments ## Summary A step-by-step guide to building a repeatable evaluation harness for local AI agents using Python, LangChain v1, Ollama, and Qwen. The harness runs test cases through a tool-calling agent, applies rule-based assertions (keyword checks, tool-call checks), and uses an LLM-as-a-judge to score outputs against a rubric. The tutorial walks through defining test cases, writing the eval script, interpreting pass/fail results, and fixing a failing test by updating the agent's system prompt. Everything runs locally with no API costs. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I evaluate a local AI agent built with LangChain and catch regressions when I change the prompt or model? Build a lightweight eval harness combining rule-based assertions with an LLM-as-a-judge. Define test cases with an input, expected keyword, expected tool call, and a judge rubric; run the agent on each case, check the keyword and tool call deterministically, then ask a separate LLM prompt to answer YES or NO against the rubric. A pass requires all three checks to succeed, giving a repeatable pass/fail summary you can rerun after any change. _daily.dev surfaces practical LangChain and agent-testing guides for developers building repeatable eval workflows._ ### Why did my LangChain agent still call a tool even after I told it in the prompt not to use tools? The underlying model can choose to follow its system instructions over an ad-hoc user instruction, or vice versa, depending on how the system prompt is written. In one documented case, an agent ignored the system prompt's default behavior and skipped tool use when a user asked it to, causing an eval to fail; adding explicit guardrails to the system prompt, such as instructing the model to never bypass tool use and to state when a tool was used, fixed the failing test without breaking the others. _track agent prompt regressions like this by following practical LangChain testing walkthroughs on daily.dev._ ### Is LLM-as-a-judge reliable enough to trust on its own for scoring AI agent outputs? No, LLM-as-a-judge should be treated as a rough guide rather than a source of truth, especially with smaller local models. A 4B parameter Qwen model used as judge was sometimes wrong when scoring agent answers against a rubric, so rule-based checks remain more reliable wherever they can be written, and a solid evaluation setup combines both rule-based assertions and an LLM judge rather than relying on the judge alone. _developers weighing LLM-as-judge tradeoffs can find hands-on agent evaluation writeups on daily.dev._ ## Similar posts on daily.dev - [How to Build a Self-Evaluating AI System: Automated Testing and Evaluation Pipelines for LLM Applications](https://daily.dev/posts/how-to-build-a-self-evaluating-ai-system-automated-testing-and-evaluation-pipelines-for-llm-applica-yppnqlvra) · freeCodeCamp · 1 upvotes · 0 comments - [LLM-as-a-Judge: What It Is, Why It Works, and How to Use It to Evaluate AI Models](https://daily.dev/posts/llm-as-a-judge-what-it-is-why-it-works-and-how-to-use-it-to-evaluate-ai-models-oxkip3mvc) · Towards Data Science · 1 upvotes · 0 comments - [How to Build Your Own Local AI Agent with Tool Calling and Memory](https://daily.dev/posts/how-to-build-your-own-local-ai-agent-with-tool-calling-and-memory-edzytv5sh) · freeCodeCamp · 3 upvotes · 0 comments - [Building Custom LLM Judges for AI Agent Accuracy](https://daily.dev/posts/building-custom-llm-judges-for-ai-agent-accuracy-phlvzjknk) · databricks · 55 upvotes · 2 comments --- Tags: [#python](https://daily.dev/tags/python), [#langchain](https://daily.dev/tags/langchain), [#ollama](https://daily.dev/tags/ollama) [View this post on daily.dev](https://daily.dev/posts/how-to-evaluate-ai-agents-with-an-llm-as-a-judge-harness-in-python-frgzj1tat) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python","url":"https://daily.dev/posts/how-to-evaluate-ai-agents-with-an-llm-as-a-judge-harness-in-python-frgzj1tat","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-to-evaluate-ai-agents-with-an-llm-as-a-judge-harness-in-python-frgzj1tat"},"datePublished":"2026-07-17T23:08:41.258Z","dateModified":"2026-10-15T02:22:52.936Z","description":"A step-by-step guide to building a repeatable evaluation harness for local AI agents using Python, LangChain v1, Ollama, and Qwen. The harness runs test cases...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4dbaa960370eaccf8600a7ecb7ac4e7c?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4dbaa960370eaccf8600a7ecb7ac4e7c?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"freeCodeCamp","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"freeCodeCamp","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1628412854/logos/freecodecamp","url":"https://daily.dev/sources/freecodecamp"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-to-evaluate-ai-agents-with-an-llm-as-a-judge-harness-in-python-frgzj1tat","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,langchain,ollama","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"freeCodeCamp","item":"https://daily.dev/sources/freecodecamp"},{"@type":"ListItem","position":3,"name":"How to Evaluate AI Agents with an LLM-as-a-Judge Harness in Python"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-to-evaluate-ai-agents-with-an-llm-as-a-judge-harness-in-python-frgzj1tat#faq","mainEntity":[{"@type":"Question","name":"How do I evaluate a local AI agent built with LangChain and catch regressions when I change the prompt or model?","acceptedAnswer":{"@type":"Answer","text":"Build a lightweight eval harness combining rule-based assertions with an LLM-as-a-judge. Define test cases with an input, expected keyword, expected tool call, and a judge rubric; run the agent on each case, check the keyword and tool call deterministically, then ask a separate LLM prompt to answer YES or NO against the rubric. A pass requires all three checks to succeed, giving a repeatable pass/fail summary you can rerun after any change. daily.dev surfaces practical LangChain and agent-testing guides for developers building repeatable eval workflows."}},{"@type":"Question","name":"Why did my LangChain agent still call a tool even after I told it in the prompt not to use tools?","acceptedAnswer":{"@type":"Answer","text":"The underlying model can choose to follow its system instructions over an ad-hoc user instruction, or vice versa, depending on how the system prompt is written. In one documented case, an agent ignored the system prompt's default behavior and skipped tool use when a user asked it to, causing an eval to fail; adding explicit guardrails to the system prompt, such as instructing the model to never bypass tool use and to state when a tool was used, fixed the failing test without breaking the others. track agent prompt regressions like this by following practical LangChain testing walkthroughs on daily.dev."}},{"@type":"Question","name":"Is LLM-as-a-judge reliable enough to trust on its own for scoring AI agent outputs?","acceptedAnswer":{"@type":"Answer","text":"No, LLM-as-a-judge should be treated as a rough guide rather than a source of truth, especially with smaller local models. A 4B parameter Qwen model used as judge was sometimes wrong when scoring agent answers against a rubric, so rule-based checks remain more reliable wherever they can be written, and a solid evaluation setup combines both rule-based assertions and an LLM judge rather than relying on the judge alone. developers weighing LLM-as-judge tradeoffs can find hands-on agent evaluation writeups on daily.dev."}}]} ```