--- title: Tempo 3.1 release: new features for Kafka, TraceQL... description: Tempo 3.1 ships community-contributed Kafka client improvements (TLS, more SASL mechanisms, rack-aware fetching, configurable producer compression),... canonical: https://daily.dev/posts/tempo-3-1-release-new-features-for-kafka-traceql-metrics-updates-trace-redaction-and-more-b7ekzaznv twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Tempo 3.1 release: new features for Kafka, TraceQL metrics updates, trace redaction, and more | daily.dev og:description: Tempo 3.1 ships community-contributed Kafka client improvements (TLS, more SASL mechanisms, rack-aware fetching, configurable producer compression),... og:url: https://daily.dev/posts/tempo-3-1-release-new-features-for-kafka-traceql-metrics-updates-trace-redaction-and-more-b7ekzaznv og:image: https://api.daily.dev/og/posts/b7eKZAZNV.png og:image:alt: Tempo 3.1 release: new features for Kafka, TraceQL metrics updates, trace redaction, and more og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Tempo 3.1 release: new features for Kafka, TraceQL metrics updates, trace redaction, and more **[Grafana Labs](https://daily.dev/sources/grafana)** · 17 min read · 39 upvotes · 5 comments ## Summary Tempo 3.1 ships community-contributed Kafka client improvements (TLS, more SASL mechanisms, rack-aware fetching, configurable producer compression), query-based trace redaction via tempo-cli, and several TraceQL metrics updates including sampling-aware extrapolation, arithmetic between metrics queries, and a faster default read path on vParquet5. It also adds metrics-generator enhancements (unmatched span kind labels, db.system.name support, lower per-span costs), span pruning for large traces, a new traces-diff API/CLI/MCP tool for comparing traces, and makes vParquet5 the default block format. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I redact specific traces in Grafana Tempo using a TraceQL query instead of listing trace IDs? Tempo 3.1 lets tempo-cli redact accept a TraceQL query via --query instead of only trace IDs, matching spans by resource.* and span.* attribute equality combined with && or ||. Run with --dry-run first to see job counts without modifying data, then rerun without it to permanently rewrite blocks; --start and --end can limit the time range, but only once every scheduler and worker in the cell runs at least 3.1. _Teams rolling out query-based redaction can track Tempo release details and gotchas on daily.dev._ ### How does Tempo 3.1's with(extrapolate=true) hint correct TraceQL metrics for sampled traces? The experimental with(extrapolate=true) hint scales span counts at query time using the sampling rate stored in each span's tracestate field, so a 50%-sampled trace reporting 1,000 matched spans is reported as 2,000. It applies to rate, count_over_time, sum_over_time, avg_over_time, histogram_over_time, quantile_over_time, and compare, but not min_over_time or max_over_time, and requires vParquet4 blocks or later. _Anyone tuning adaptive sampling pipelines can follow tracing tool updates like this on daily.dev._ ### Why did database service nodes disappear from my Tempo service graph after upgrading OpenTelemetry instrumentation? OpenTelemetry semantic conventions v1.30.0 renamed the db.system attribute to db.system.name, and instrumentation emitting only the new name stopped being recognized as a database call in Tempo versions before 3.1, silently dropping those nodes from the service map. Tempo 3.1 recognizes both attributes, preferring db.system when both are present, so upgrading SDKs no longer breaks existing dashboards and alerts. _Engineers debugging service-map regressions after OTel upgrades can keep up with fixes like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@nahum** · 2 upvotes > traces-diff is the one I want. half my incident time is "what's different between the slow request and the normal one", done by eye in two tabs. > > does it align spans by name or by tree structure? retries and fan-out would make naive matching messy **@ristotoldsep** · 2 upvotes > Trace redaction is the boring feature that makes this actually usable at work. nice. **@kartiknvj** · 1 upvotes > The traces-diff tool is the feature I did not know I needed. For agent runs, comparing two traces of the same task is usually how I find where a run silently diverged, and doing that by eye across spans is painful. Does the query-based redaction run before the diff, so I can strip prompt payloads and still compare structure? **@ahmetozel** · 0 upvotes > The fallback that counts spans without sampling metadata as one is operationally convenient, but it should not be interpreted as proving those spans were unsampled. During partial adoption, I would display the share of spans with a usable sampling rate next to the extrapolated result. Otherwise a rollout can look like a traffic change when only metadata coverage changed. > > This matters especially for the error-rate expression. If error traces and successful traces have different retention probabilities, the numerator and denominator need their own valid weights; a single service-wide sampling... **@andro161** · 0 upvotes > In the context of the Tempo 3.1 release, what specific performance optimization was introduced to the default read path for metrics, and which block format was made the default across the system? ## Similar posts on daily.dev - [Tempo 3.0 release: a new architecture for scale and lower TCO, TraceQL metrics GA, and more](https://daily.dev/posts/tempo-3-0-release-a-new-architecture-for-scale-and-lower-tco-traceql-metrics-ga-and-more-ubfo6o7ia) · Grafana Labs · 50 upvotes · 1 comments - [Grafana Tempo 3.1 Adds Query-Based Trace Redaction and Sampling-Aware TraceQL Metrics](https://daily.dev/posts/grafana-tempo-3-1-adds-query-based-trace-redaction-and-sampling-aware-traceql-metrics-sniios6sj) · InfoQ · 4 upvotes · 0 comments - [Grafana Tempo 2.9 release: MCP server support, TraceQL metrics sampling, and more](https://daily.dev/posts/grafana-tempo-2-9-release-mcp-server-support-traceql-metrics-sampling-and-more-1fkybrkfj) · Grafana Labs · 23 upvotes · 0 comments - [Grafana Tempo: Setup, Configuration, and Best Practices](https://daily.dev/posts/grafana-tempo-setup-configuration-and-best-practices-grrxdgmhd) · Last9 · 0 upvotes · 0 comments --- Tags: [#devops](https://daily.dev/tags/devops), [#observability](https://daily.dev/tags/observability), [#kafka](https://daily.dev/tags/kafka) [View this post on daily.dev](https://daily.dev/posts/tempo-3-1-release-new-features-for-kafka-traceql-metrics-updates-trace-redaction-and-more-b7ekzaznv) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Tempo 3.1 release: new features for Kafka, TraceQL metrics updates, trace redaction, and more","url":"https://daily.dev/posts/tempo-3-1-release-new-features-for-kafka-traceql-metrics-updates-trace-redaction-and-more-b7ekzaznv","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/tempo-3-1-release-new-features-for-kafka-traceql-metrics-updates-trace-redaction-and-more-b7ekzaznv"},"datePublished":"2026-10-01T07:23:00.780Z","dateModified":"2026-10-01T07:23:23.828Z","description":"Tempo 3.1 ships community-contributed Kafka client improvements (TLS, more SASL mechanisms, rack-aware fetching, configurable producer compression),...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/75e6aca49705ddf77fd6a8eea19684d3?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/75e6aca49705ddf77fd6a8eea19684d3?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Grafana Labs","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Grafana Labs","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/769cb8df467e4c428abd75e0c2491d1d","url":"https://daily.dev/sources/grafana"},"commentCount":5,"discussionUrl":"https://daily.dev/posts/tempo-3-1-release-new-features-for-kafka-traceql-metrics-updates-trace-redaction-and-more-b7ekzaznv","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":39},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":5}],"keywords":"devops,observability,kafka","timeRequired":"PT17M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Grafana Labs","item":"https://daily.dev/sources/grafana"},{"@type":"ListItem","position":3,"name":"Tempo 3.1 release: new features for Kafka, TraceQL metrics updates, trace redaction, and more"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/tempo-3-1-release-new-features-for-kafka-traceql-metrics-updates-trace-redaction-and-more-b7ekzaznv","comment":[{"@type":"Comment","text":"traces-diff is the one I want. half my incident time is “what’s different between the slow request and the normal one”, done by eye in two tabs.\ndoes it align spans by name or by tree structure? retries and fan-out would make naive matching messy","datePublished":"2026-10-01T17:32:51.649Z","url":"https://daily.dev/posts/b7eKZAZNV#c-KlYAe70Wz","author":{"@type":"Person","name":"Nahum Litvin","url":"https://daily.dev/nahum","image":"https://lh3.googleusercontent.com/a-/AOh14GhvJ2y-9R45d5ee78ZdAo8l7J3t4G8fINkx5PEqBfA=s100"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"Trace redaction is the boring feature that makes this actually usable at work. nice.","datePublished":"2026-10-02T11:40:58.394Z","url":"https://daily.dev/posts/b7eKZAZNV#c-APeZVBax8","author":{"@type":"Person","name":"Risto Tõldsep","url":"https://daily.dev/ristotoldsep","image":"https://lh3.googleusercontent.com/a/ACg8ocLDWc6mZn0JwNmXXw6WY0L_HJ6pegzRttooC5VgtXESHSj3MNYx=s96-c"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"The traces-diff tool is the feature I did not know I needed. For agent runs, comparing two traces of the same task is usually how I find where a run silently diverged, and doing that by eye across spans is painful. Does the query-based redaction run before the diff, so I can strip prompt payloads and still compare structure?","datePublished":"2026-10-01T18:55:07.808Z","url":"https://daily.dev/posts/b7eKZAZNV#c-yVamL0Ae1","author":{"@type":"Person","name":"kartik-nvjk","url":"https://daily.dev/kartiknvj","image":"https://media.daily.dev/image/upload/s--3gGgsVCw--/f_auto/v1781456774/avatars/avatar_TvTVeiMdkRCqWUDullFmy?_a=BAMAMiWQ0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"The fallback that counts spans without sampling metadata as one is operationally convenient, but it should not be interpreted as proving those spans were unsampled. During partial adoption, I would display the share of spans with a usable sampling rate next to the extrapolated result. Otherwise a rollout can look like a traffic change when only metadata coverage changed.\nThis matters especially for the error-rate expression. If error traces and successful traces have different retention probabilities, the numerator and denominator need their own valid weights; a single service-wide sampling factor will not cancel the bias. Comparing against independently collected request counters would help distinguish extrapolation behavior from an actual change in service traffic.","datePublished":"2026-10-06T18:38:16.751Z","url":"https://daily.dev/posts/b7eKZAZNV#c-eB4S0aQqi","author":{"@type":"Person","name":"Ahmet Özel","url":"https://daily.dev/ahmetozel","image":"https://avatars.githubusercontent.com/u/70992231?v=4"}},{"@type":"Comment","text":"In the context of the Tempo 3.1 release, what specific performance optimization was introduced to the default read path for metrics, and which block format was made the default across the system?","datePublished":"2026-10-11T00:28:09.707Z","url":"https://daily.dev/posts/b7eKZAZNV#c-uTEEVx64q","author":{"@type":"Person","name":"Andromeda","url":"https://daily.dev/andro161","image":"https://media.daily.dev/image/upload/s--FczMGWTg--/f_auto/v1791677381/avatars/avatar_3RIxldxMFpzm5fIHIGMl5?_a=BAMAMicg0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/tempo-3-1-release-new-features-for-kafka-traceql-metrics-updates-trace-redaction-and-more-b7ekzaznv#faq","mainEntity":[{"@type":"Question","name":"How do I redact specific traces in Grafana Tempo using a TraceQL query instead of listing trace IDs?","acceptedAnswer":{"@type":"Answer","text":"Tempo 3.1 lets tempo-cli redact accept a TraceQL query via --query instead of only trace IDs, matching spans by resource.* and span.* attribute equality combined with && or ||. Run with --dry-run first to see job counts without modifying data, then rerun without it to permanently rewrite blocks; --start and --end can limit the time range, but only once every scheduler and worker in the cell runs at least 3.1. Teams rolling out query-based redaction can track Tempo release details and gotchas on daily.dev."}},{"@type":"Question","name":"How does Tempo 3.1's with(extrapolate=true) hint correct TraceQL metrics for sampled traces?","acceptedAnswer":{"@type":"Answer","text":"The experimental with(extrapolate=true) hint scales span counts at query time using the sampling rate stored in each span's tracestate field, so a 50%-sampled trace reporting 1,000 matched spans is reported as 2,000. It applies to rate, count_over_time, sum_over_time, avg_over_time, histogram_over_time, quantile_over_time, and compare, but not min_over_time or max_over_time, and requires vParquet4 blocks or later. Anyone tuning adaptive sampling pipelines can follow tracing tool updates like this on daily.dev."}},{"@type":"Question","name":"Why did database service nodes disappear from my Tempo service graph after upgrading OpenTelemetry instrumentation?","acceptedAnswer":{"@type":"Answer","text":"OpenTelemetry semantic conventions v1.30.0 renamed the db.system attribute to db.system.name, and instrumentation emitting only the new name stopped being recognized as a database call in Tempo versions before 3.1, silently dropping those nodes from the service map. Tempo 3.1 recognizes both attributes, preferring db.system when both are present, so upgrading SDKs no longer breaks existing dashboards and alerts. Engineers debugging service-map regressions after OTel upgrades can keep up with fixes like this on daily.dev."}}]} ```