LLM Evals Are Based on Vibes — I Built the Missing Layer That Decides What Ships
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Most LLM evaluation systems rely on a single composite score and human judgment, which fails to catch confident hallucinations that pass threshold checks. This post presents a lightweight Python evaluation layer that splits faithfulness into two separate signals — attribution (grounding in context) and specificity (concreteness of claims). High specificity combined with low attribution is the hallmark of a confident hallucination. The system adds a decision layer that converts scores into actionable verdicts (ACCEPT, REVIEW, REJECT) with plain-English reasons and next actions. It runs locally using sentence-transformers in ~291ms with no API calls by default, escalating to an LLM judge only for borderline scores (0.45–0.65). A disagreement signal (standard deviation across dimension scores) routes high-variance outputs to human review. A regression suite integrates with CI/CD to block deployments when prompt changes degrade quality. Limitations include inability to detect implicit semantic drift or cross-document contradictions.
