--- title: How to turn slow queries into actionable reliability... description: A practical guide walks through building a progressively more sophisticated observability workflow for slow SQL queries using OpenTelemetry. It starts with a... canonical: https://daily.dev/posts/how-to-turn-slow-queries-into-actionable-reliability-metrics-with-opentelemetry-5crln7qq6 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How to turn slow queries into actionable reliability metrics with OpenTelemetry | daily.dev og:description: A practical guide walks through building a progressively more sophisticated observability workflow for slow SQL queries using OpenTelemetry. It starts with a... og:url: https://daily.dev/posts/how-to-turn-slow-queries-into-actionable-reliability-metrics-with-opentelemetry-5crln7qq6 og:image: https://api.daily.dev/og/posts/5crLN7qq6.png og:image:alt: How to turn slow queries into actionable reliability metrics with OpenTelemetry og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How to turn slow queries into actionable reliability metrics with OpenTelemetry **[CNCF](https://daily.dev/sources/cncf)** · 12 min read · 0 upvotes · 0 comments ## Summary A practical guide walks through building a progressively more sophisticated observability workflow for slow SQL queries using OpenTelemetry. It starts with a basic dashboard sorting queries by duration, then adds a traffic-weighted impact score (average duration times call count) to prioritize optimization work, and finally introduces anomaly detection via the spanmetrics connector and Prometheus adaptive baselines to catch queries that deviate from normal behavior. A companion lab (Go app, PostgreSQL, docker-otel-lgtm stack) lets readers reproduce the setup with Docker. Production considerations covered include metric cardinality explosion from raw SQL in labels, redacting sensitive data at the Collector, and the 24-48 hour warm-up period anomaly baselines need. The piece closes by noting that anomaly detection only surfaces symptoms, not root causes, and points to Causely's causal modeling as the next step. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I prioritize which slow SQL queries to optimize first? Prioritize by traffic-weighted impact rather than raw average duration: multiply average query duration by call count to get an impact score. A 150ms query run 10,000 times has far more aggregate impact than a 2.3-second query run only 5 times, even though the latter looks slower on paper. Sorting by this impact score surfaces the queries that matter most to users. _Track prioritization techniques like traffic-weighted impact scoring for database performance work on daily.dev._ ### How can I detect when a database query has become abnormally slow instead of just slow? Use adaptive anomaly detection baselines built from historical latency data via Prometheus recording rules, such as Grafana's PromQL Anomaly Detection framework. These rules compute a smoothed baseline plus upper and lower bands (baseline plus or minus N standard deviations); when current latency crosses the band, that's flagged as an anomaly. A query that's always been 450ms won't trigger an alert at 450ms, but a normally 50ms query spiking to 200ms will, distinguishing known slowness from new regressions. _Developers building incident-response dashboards can follow anomaly detection patterns like this on daily.dev._ ### How do I avoid exploding metric cardinality when generating span-based metrics from SQL queries? Avoid putting raw literal SQL into metric labels, since a query like SELECT * FROM orders WHERE customer_id = 12345 would create a separate time series per customer id, overwhelming the metrics backend. Instead, use prepared statements so instrumentation captures query templates rather than literal values, normalize query text, or configure the aggregation_cardinality_limit setting in the OpenTelemetry Collector's spanmetrics connector. _Anyone wiring spanmetrics into production can check cardinality pitfalls like this via daily.dev._ ## Similar posts on daily.dev - [Optimizing queries by using observability](https://daily.dev/posts/optimizing-queries-by-using-observability-yrpuktkuk) · InfoWorld · 0 upvotes · 0 comments - [7 Observability Solutions for Full-Fidelity Telemetry](https://daily.dev/posts/7-observability-solutions-for-full-fidelity-telemetry-z4sznixmd) · Last9 · 0 upvotes · 0 comments - [A Guide to Web Application Monitoring](https://daily.dev/posts/a-guide-to-web-application-monitoring-aowmrvcdn) · RUBYLAND · 0 upvotes · 0 comments - [How to build a cost-effective observability platform with OpenTelemetry](https://daily.dev/posts/how-to-build-a-cost-effective-observability-platform-with-opentelemetry-z8pbichl4) · CNCF · 8 upvotes · 0 comments - [Observability on Kubernetes: OpenTelemetry, Camel & Quarkus](https://daily.dev/posts/observability-on-kubernetes-opentelemetry-camel-quarkus-q9gx7ah7s) · Piotr Minkowski TechBlog · 5 upvotes · 1 comments --- Tags: [#postgresql](https://daily.dev/tags/postgresql), [#observability](https://daily.dev/tags/observability), [#opentelemetry](https://daily.dev/tags/opentelemetry), [#grafana](https://daily.dev/tags/grafana) [View this post on daily.dev](https://daily.dev/posts/how-to-turn-slow-queries-into-actionable-reliability-metrics-with-opentelemetry-5crln7qq6) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How to turn slow queries into actionable reliability metrics with OpenTelemetry","url":"https://daily.dev/posts/how-to-turn-slow-queries-into-actionable-reliability-metrics-with-opentelemetry-5crln7qq6","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-to-turn-slow-queries-into-actionable-reliability-metrics-with-opentelemetry-5crln7qq6"},"datePublished":"2026-08-21T11:04:14.600Z","dateModified":"2026-09-13T18:44:50.658Z","description":"A practical guide walks through building a progressively more sophisticated observability workflow for slow SQL queries using OpenTelemetry. It starts with a...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f8665bbaad739565d6916f4d0cbf4313?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f8665bbaad739565d6916f4d0cbf4313?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"CNCF","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"CNCF","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/55536a03390741eebd96d2f9b74d2d8d","url":"https://daily.dev/sources/cncf"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-to-turn-slow-queries-into-actionable-reliability-metrics-with-opentelemetry-5crln7qq6","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"postgresql,observability,opentelemetry,grafana","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"CNCF","item":"https://daily.dev/sources/cncf"},{"@type":"ListItem","position":3,"name":"How to turn slow queries into actionable reliability metrics with OpenTelemetry"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-to-turn-slow-queries-into-actionable-reliability-metrics-with-opentelemetry-5crln7qq6#faq","mainEntity":[{"@type":"Question","name":"How do I prioritize which slow SQL queries to optimize first?","acceptedAnswer":{"@type":"Answer","text":"Prioritize by traffic-weighted impact rather than raw average duration: multiply average query duration by call count to get an impact score. A 150ms query run 10,000 times has far more aggregate impact than a 2.3-second query run only 5 times, even though the latter looks slower on paper. Sorting by this impact score surfaces the queries that matter most to users. Track prioritization techniques like traffic-weighted impact scoring for database performance work on daily.dev."}},{"@type":"Question","name":"How can I detect when a database query has become abnormally slow instead of just slow?","acceptedAnswer":{"@type":"Answer","text":"Use adaptive anomaly detection baselines built from historical latency data via Prometheus recording rules, such as Grafana's PromQL Anomaly Detection framework. These rules compute a smoothed baseline plus upper and lower bands (baseline plus or minus N standard deviations); when current latency crosses the band, that's flagged as an anomaly. A query that's always been 450ms won't trigger an alert at 450ms, but a normally 50ms query spiking to 200ms will, distinguishing known slowness from new regressions. Developers building incident-response dashboards can follow anomaly detection patterns like this on daily.dev."}},{"@type":"Question","name":"How do I avoid exploding metric cardinality when generating span-based metrics from SQL queries?","acceptedAnswer":{"@type":"Answer","text":"Avoid putting raw literal SQL into metric labels, since a query like SELECT * FROM orders WHERE customer_id = 12345 would create a separate time series per customer id, overwhelming the metrics backend. Instead, use prepared statements so instrumentation captures query templates rather than literal values, normalize query text, or configure the aggregation_cardinality_limit setting in the OpenTelemetry Collector's spanmetrics connector. Anyone wiring spanmetrics into production can check cardinality pitfalls like this via daily.dev."}}]} ```