--- title: "Control trace volume with OpenTelemetry tail-based sampling" url: https://daily.dev/posts/control-trace-volume-with-opentelemetry-tail-based-sampling-r7d6ugtol source_url: https://www.datadoghq.com/blog/control-trace-volume-with-opentelemetry-tail-based-sampling type: article source: "Datadog" published: 2026-08-21T15:25:34.699Z updated: 2026-08-21T15:35:56.701Z tags: ["devops", "observability", "opentelemetry", "apm"] reading_time: 11 upvotes: 2 comments: 1 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Control trace volume with OpenTelemetry tail-based sampling **[Datadog](https://daily.dev/sources/datadog)** · 11 min read · 2 upvotes · 1 comments ## Summary A detailed guide walks through configuring tail-based sampling in the OpenTelemetry Collector to reduce trace volume while preserving errors, slow requests, and other high-value traces. It explains the gateway deployment pattern needed for tail sampling, how Span Metrics are computed before sampling to keep service health accurate, and how to layer probabilistic, error, latency, heavy-hitter, and drop policies. A rideshare demo app shows a 98% reduction in exported trace volume (1.87 million traces down to about 27,000) while Span Metrics stay fully representative. It also covers developer experience fixes like always-sample-dev and force-sample-on-demand policies, plus memory tuning and operational guidance. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I configure tail-based sampling in the OpenTelemetry Collector to reduce trace volume without losing important traces? Use the tail_sampling processor in a gateway deployment where all spans for a trace route to the same collector via the Load Balancing exporter. Layer policies from most specific to least: retain 100% of error traces, retain traces above a latency threshold (e.g. 750ms), retain critical paths, sample remaining healthy traffic at a low rate like 5%, and explicitly drop low-value traffic like health checks. This approach reduced trace volume by about 98% in a rideshare demo (1.87 million traces to about 27,000) while keeping service health metrics intact. _daily.dev surfaces practical guides like this for engineers tuning observability pipelines and controlling APM costs._ ### Why does the OpenTelemetry Collector tail_sampling processor require a gateway deployment pattern? All spans belonging to a single trace must reach the same collector instance so it can group them by trace ID and evaluate sampling policies against the complete trace. A gateway pattern solves this with upstream collectors that forward spans over OTLP using the Load Balancing exporter, which routes spans sharing a trace ID to the same gateway instance, where policies are then applied and Span Metrics computed. _Teams designing tracing architecture can find this kind of infrastructure detail through daily.dev._ ### How does tail-based sampling differ from head-based sampling in distributed tracing? Head-based sampling decides whether to keep a trace at the start, before it completes, so the SDK's root-span decision can miss errors or latency spikes that occur deep in the call stack. Tail-based sampling waits until the trace completes, buffering all its spans in the collector, then evaluates status codes, latency, and attributes to decide what to keep, at the cost of higher memory usage. _daily.dev helps engineers compare tracing strategies like head- versus tail-based sampling before committing to one._ ## Community discussion Top comments from developers on daily.dev. **@kartiknvj** · 1 upvotes > Tail-based sampling is the right call for agent traces because the interesting spans are usually the slow or failed ones you would drop with head sampling. I keep every errored trace and sample the happy path, which cut volume hard without losing the debuggable cases. What is your retention split between error and normal traces? ## Similar posts on daily.dev - [How Capital One Cut Tracing Data by 70% With OpenTelemetry](https://daily.dev/posts/how-capital-one-cut-tracing-data-by-70-with-opentelemetry-homzqkwiv) · The New Stack · 0 upvotes · 0 comments - [Sampling: the philosopher’s stone of distributed tracing](https://daily.dev/posts/sampling-the-philosopher-s-stone-of-distributed-tracing-gpotwuue0) · The New Stack · 0 upvotes · 0 comments --- Tags: [#devops](https://daily.dev/tags/devops), [#observability](https://daily.dev/tags/observability), [#opentelemetry](https://daily.dev/tags/opentelemetry), [#apm](https://daily.dev/tags/apm) [View this post on daily.dev](https://daily.dev/posts/control-trace-volume-with-opentelemetry-tail-based-sampling-r7d6ugtol)