--- title: "How to evaluate LLMs before production" url: https://daily.dev/posts/how-to-evaluate-llms-before-production-ps4g4zhbm source_url: https://github.blog/ai-and-ml/llms/how-to-evaluate-llms-before-production type: article source: "GitHub Blog" published: 2026-08-25T21:39:12.592Z updated: 2026-08-25T21:58:22.720Z tags: ["security", "llm", "github", "appsec"] reading_time: 13 upvotes: 1 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How to evaluate LLMs before production **[GitHub Blog](https://daily.dev/sources/ghblog)** · 13 min read · 1 upvotes · 0 comments ## Summary GitHub engineers share practical lessons learned evaluating an LLM system for reducing false positives in secret scanning while preserving recall for security safety. Key practices include anchoring evaluation to a product decision rather than model tweaks, treating offline evaluation like integration testing with versioned prompts/models, changing one variable at a time, keeping offline evaluation close to production conditions, treating production labels as noisy signals rather than ground truth, using synthetic data to fill coverage gaps, performing error analysis to categorize failure sources, and using LLM-as-judge to triage human review. The team achieved a 95% reduction in false positives while keeping recall within a defined guardrail on their offline dataset. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much did GitHub reduce false positives in secret scanning using an LLM-based evaluation approach? GitHub's applied science team achieved a 95% reduction in false positives on their evaluated offline dataset for secret scanning, while keeping recall within a predefined safety guardrail. This was reached through repeated offline evaluation, error analysis, and treating precision improvements as secondary to maintaining recall as a safety constraint, since incorrectly suppressing a real credential was considered more consequential than an extra alert. _Teams tuning precision-recall tradeoffs in security workflows can track LLM evaluation practices like this on daily.dev._ ### Why shouldn't I treat production labels as ground truth when evaluating an LLM system? Production labels often reflect workflow outcomes rather than reliable ground truth, because a dismissed or resolved alert doesn't necessarily mean it was a false positive. A developer might resolve a secret-scanning alert because the credential was rotated, the risk was accepted, the alert blocked a workflow, or it truly was misclassified, so these different outcomes can look identical in the data while representing different underlying truths. _Anyone building evaluation pipelines for ML systems can follow data-quality practices like this on daily.dev._ ### What's a safe pattern for using LLM-as-judge in an evaluation pipeline? A safer pattern treats the judge's output as another prediction rather than ground truth, using it for triage: automatically process clear low-risk cases, route low-confidence or high-impact cases to human reviewers, periodically sample high-confidence cases to check for systematic errors, track disagreement between judge, system, and reviewers, and version the judge prompt like any other model component. _Developers designing human-in-the-loop review workflows can find approaches like this on daily.dev._ ## Similar posts on daily.dev - [AI Evaluation Engineering: Build a Production-Grade LLM Evaluation Platform from Scratch \[Full Handbook\]](https://daily.dev/posts/ai-evaluation-engineering-build-a-production-grade-llm-evaluation-platform-from-scratch-full-handb-ju9v0rogq) · freeCodeCamp · 5 upvotes · 0 comments - [Integrating LLM Evaluations into Your CI Pipeline: Best Practices for AI Teams](https://daily.dev/posts/integrating-llm-evaluations-into-your-ci-pipeline-best-practices-for-ai-teams-kg95szbwx) · PromptLayer Blog · 0 upvotes · 0 comments - [Beyond LLM-as-a-judge: Establishing LLM evaluations as a foundation for trustworthy agentic AI systems](https://daily.dev/posts/beyond-llm-as-a-judge-establishing-llm-evaluations-as-a-foundation-for-trustworthy-agentic-ai-syste-m8kymtr9x) · Dynatrace · 1 upvotes · 0 comments - [Choosing Effective LLM Evaluation Strategies for AI Teams](https://daily.dev/posts/choosing-effective-llm-evaluation-strategies-for-ai-teams-zboitrqp2) · PromptLayer Blog · 0 upvotes · 0 comments --- Tags: [#security](https://daily.dev/tags/security), [#llm](https://daily.dev/tags/llm), [#github](https://daily.dev/tags/github), [#appsec](https://daily.dev/tags/appsec) [View this post on daily.dev](https://daily.dev/posts/how-to-evaluate-llms-before-production-ps4g4zhbm)