--- title: "How to turn slow queries into actionable reliability metrics with OpenTelemetry" url: https://daily.dev/posts/how-to-turn-slow-queries-into-actionable-reliability-metrics-with-opentelemetry-5crln7qq6 source_url: https://www.cncf.io/blog/2026/08/21/how-to-turn-slow-queries-into-actionable-reliability-metrics-with-opentelemetry type: article source: "CNCF" published: 2026-08-21T11:04:14.600Z updated: 2026-08-21T11:04:44.696Z tags: ["postgresql", "observability", "opentelemetry", "grafana"] reading_time: 12 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How to turn slow queries into actionable reliability metrics with OpenTelemetry **[CNCF](https://daily.dev/sources/cncf)** · 12 min read · 0 upvotes · 0 comments ## Summary A practical guide walks through building a progressively more sophisticated observability workflow for slow SQL queries using OpenTelemetry. It starts with a basic dashboard sorting queries by duration, then adds a traffic-weighted impact score (average duration times call count) to prioritize optimization work, and finally introduces anomaly detection via the spanmetrics connector and Prometheus adaptive baselines to catch queries that deviate from normal behavior. A companion lab (Go app, PostgreSQL, docker-otel-lgtm stack) lets readers reproduce the setup with Docker. Production considerations covered include metric cardinality explosion from raw SQL in labels, redacting sensitive data at the Collector, and the 24-48 hour warm-up period anomaly baselines need. The piece closes by noting that anomaly detection only surfaces symptoms, not root causes, and points to Causely's causal modeling as the next step. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do you calculate which slow SQL queries are actually worth optimizing first? Multiply average query duration by call count to get an impact score, then sort queries by that score instead of by raw duration. A query averaging 150ms but running 10,000 times has far more total user impact than one averaging 2.3s but running only 5 times, so impact-weighted sorting surfaces the queries that matter most for optimization prioritization. _daily.dev surfaces engineering write-ups like this for teams deciding how to prioritize database optimization work._ ### How can I detect anomalous database query latency instead of relying on fixed thresholds? Use the OpenTelemetry Collector's spanmetrics connector to turn database spans into latency histograms labeled by service, db.system, and query text, store them in a Prometheus-compatible backend like Mimir, then apply Grafana's PromQL Anomaly Detection recording rules to compute a smoothed baseline plus upper and lower standard-deviation bands. Latency exceeding those bands flags a real deviation from normal behavior rather than a static threshold breach. _track approaches like adaptive anomaly baselines on daily.dev when building your own incident response tooling._ ### Why does putting raw SQL text into OpenTelemetry metric labels cause problems in production? Raw SQL in metric labels causes cardinality explosion because a query like SELECT * FROM orders WHERE customer_id = 12345 generates a separate metric series per literal value, such as per customer id. The fix is to use prepared statements so instrumentation captures query templates instead of literals, normalize query text, or set aggregation_cardinality_limit in the spanmetrics connector. _developers rolling out span metrics in production can compare cardinality pitfalls like this on daily.dev._ ## Similar posts on daily.dev - [Optimizing queries by using observability](https://daily.dev/posts/optimizing-queries-by-using-observability-yrpuktkuk) · InfoWorld · 0 upvotes · 0 comments - [One Database for LLM Observability: Traces, Metrics, and Conversations](https://daily.dev/posts/one-database-for-llm-observability-traces-metrics-and-conversations-wrkkswkjl) · ITNEXT · 0 upvotes · 0 comments --- Tags: [#postgresql](https://daily.dev/tags/postgresql), [#observability](https://daily.dev/tags/observability), [#opentelemetry](https://daily.dev/tags/opentelemetry), [#grafana](https://daily.dev/tags/grafana) [View this post on daily.dev](https://daily.dev/posts/how-to-turn-slow-queries-into-actionable-reliability-metrics-with-opentelemetry-5crln7qq6)