---
title: "I shipped an AI Agent. It broke in 3 days. Here’s the eval I should have written first."
url: https://daily.dev/posts/i-shipped-an-ai-agent-it-broke-in-3-days-here-s-the-eval-i-should-have-written-first--pq4qs5n7v
source_url: https://levelup.gitconnected.com/i-shipped-an-ai-agent-it-broke-in-3-days-heres-the-eval-i-should-have-written-first-0fb8b379ffbc
type: article
source: "gitconnected"
published: 2026-08-06T14:34:51.802Z
updated: 2026-08-06T14:35:18.389Z
tags: ["llm", "ai-agents", "observability"]
reading_time: 16
upvotes: 7
comments: 2
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# I shipped an AI Agent. It broke in 3 days. Here’s the eval I should have written first.

**[gitconnected](https://daily.dev/sources/gc)** · 16 min read · 7 upvotes · 2 comments

## Summary

A junior AI engineer shares a hard-learned lesson from shipping an AI support-routing agent that broke within three days of production. The post explains the critical gap between observability (seeing what an agent did) and evaluation (proving it did the right thing), backed by LangChain survey data showing only 52% of teams have offline evals. Three common failure modes are covered: confident hallucinations, wrong tool selection with correct-looking output, and prompt drift across versions. A practical ~50-line Python eval pattern is provided using hard checks for ground truth cases and LLM-as-judge for subjective quality. The post also covers LLM-as-judge pitfalls (position bias, verbosity bias, rubric drift) and recommends calibrating judges against 20–30 human-labeled examples. Frameworks like Langfuse, DeepEval, LangSmith, and MLflow are briefly compared.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://levelup.gitconnected.com/i-shipped-an-ai-agent-it-broke-in-3-days-heres-the-eval-i-should-have-written-first-0fb8b379ffbc>

## Community discussion

Top comments from developers on daily.dev.

**@kartiknvj** · 1 upvotes

> The observability-versus-evaluation distinction is the one I wish I'd internalized earlier too: dashboards tell you what the agent did, not whether it was right. The fix that stuck for me was writing the eval as a gate in CI before shipping, so regressions fail the build instead of surfacing as a user complaint three days later. I put the setup I use here in case it saves you the same lesson: [https://dev.to/kartik-nvjk/how-i-set-up-rag-evals-in-cicd-so-they-actually-catch-regressions-46hb](https://dev.to/kartik-nvjk/how-i-set-up-rag-evals-in-cicd-so-they-actually-catch-regressions-46hb)

**@thomaswolgast** · 0 upvotes

> Very important article! But what I don't understand is this: "Respond with just 1 or 0, then a one-line reason."
>
> Are structured outputs not used anymore in practice? Or is this dumbed down for simplicity? Structured outputs are so useful and I'm skeptical about these simplified examples, because they provide bad examples.

## Similar posts on daily.dev

- [Evaluating AI Agents in Practice: Benchmarks, Frameworks, and Lessons Learned](https://daily.dev/posts/evaluating-ai-agents-in-practice-benchmarks-frameworks-and-lessons-learned-tohpewvfd) · InfoQ · 2 upvotes · 0 comments
- [AI Evals for Solo Developers 2026: A Practical Guide](https://daily.dev/posts/ai-evals-for-solo-developers-2026-a-practical-guide-ms6ijzdsf) · Alex CloudStar · 0 upvotes · 0 comments

---

Tags: [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#observability](https://daily.dev/tags/observability)

[View this post on daily.dev](https://daily.dev/posts/i-shipped-an-ai-agent-it-broke-in-3-days-here-s-the-eval-i-should-have-written-first--pq4qs5n7v)