--- title: "Evaluate LLM and agent quality in Dynatrace AI Observability with dt-evals" url: https://daily.dev/posts/evaluate-llm-and-agent-quality-in-dynatrace-ai-observability-with-dt-evals-dkozgp7zn source_url: https://www.dynatrace.com/news/blog/evaluate-llm-and-agent-quality-in-dynatrace-ai-observability type: article source: "Dynatrace" published: 2026-06-11T19:49:54.424Z updated: 2026-06-11T19:50:24.825Z tags: ["devops", "llm", "ai-agents"] reading_time: 13 upvotes: 1 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Evaluate LLM and agent quality in Dynatrace AI Observability with dt-evals **[Dynatrace](https://daily.dev/sources/dynatrace)** · 13 min read · 1 upvotes · 0 comments ## Summary dt-evals is an open source CLI tool from Dynatrace that enables teams to evaluate LLM and AI agent output quality by pulling real GenAI traces, scoring them with an LLM-as-judge approach, and writing structured results back into Dynatrace AI Observability. It supports both offline (pre-release CI/CD) and online (post-deployment sampling) evaluation modes. Built-in evaluators cover 15 quality and safety dimensions including faithfulness, hallucination, relevance, toxicity, PII leakage, and prompt injection, with support for custom evaluators. Results integrate with Dynatrace dashboards, DQL queries, and alerting workflows, enabling teams to trend quality scores, detect regressions, and gate releases on AI quality metrics alongside traditional latency and error rate signals. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [LLM Evaluation and AI Observability for Agent Monitoring](https://daily.dev/posts/llm-evaluation-and-ai-observability-for-agent-monitoring-dcaarbtie) · JetBrains · 1 upvotes · 0 comments - [Define, run, and scale custom LLM-as-a-judge evaluations in Datadog](https://daily.dev/posts/define-run-and-scale-custom-llm-as-a-judge-evaluations-in-datadog-6208qkgdt) · Datadog · 1 upvotes · 0 comments - [Debug and evaluate your AI app from your coding agent with Datadog Agent Observability](https://daily.dev/posts/debug-and-evaluate-your-ai-app-from-your-coding-agent-with-datadog-agent-observability-uwbwi17jy) · Datadog · 0 upvotes · 0 comments --- Tags: [#devops](https://daily.dev/tags/devops), [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents) [View this post on daily.dev](https://daily.dev/posts/evaluate-llm-and-agent-quality-in-dynatrace-ai-observability-with-dt-evals-dkozgp7zn)