Tail Sample

Analytics State Management

Synopsis

Buffers OTLP spans by trace ID and makes one keep-or-drop decision for the whole trace, once enough of it has arrived.

Schema

- tail_sample:
    decision_wait_sec: <numeric>
    destination: <string>
    policies:
      - type: <enum>
        threshold_ms: <numeric>
        expression: <script>
        percent: <numeric>
    max_traces: <numeric>
    max_spans_per_trace: <numeric>
    max_buffer_bytes: <numeric>
    decision_cache_sec: <numeric>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration

Field	Required	Default	Description
destination	Y	-	Named target of the route that receives the kept traces
policies	Y	-	Keep policies, evaluated in order. At least one is required
decision_wait_sec	N	10	Seconds to buffer a trace before deciding
max_traces	N	50000	Traces buffered concurrently
max_spans_per_trace	N	512	Spans retained per trace
max_buffer_bytes	N	268435456 (256 MiB)	Total buffer ceiling
decision_cache_sec	N	300	How long a decision is remembered, so late spans of a decided trace follow it
description	N	-	Explanatory note
if	N	-	Condition to run
ignore_failure	N	false	See Handling Failures. This does not mean "keep filtering and ignore errors" — see the warning below.
on_failure	N	-	See Handling Failures
on_success	N	-	See Handling Success
tag	N	-	Identifier
disabled	N	false	When true, the processor is skipped and the event continues to the next one. Lets you take a processor out of the path without removing its configuration

warning

ignore_failure: true silently turns this processor off. The drop is signalled to the pipeline as an error value, and the ignore_failure check runs before that value is inspected — so a matched event is kept instead of dropped, no error is logged, and the pipeline reports success. The processor appears to run normally while filtering nothing.

Use it only if you genuinely want a pass-through. To tolerate real errors without losing the filtering, leave ignore_failure unset and handle the failure with on_failure instead.

Policy

Field	Required	Default	Description
type	Y	-	error, latency, filter or probabilistic
threshold_ms	Y*	-	Trace duration at or above which to keep. Required by latency, and must be positive
expression	Y*	-	Condition selecting traces to keep. Required by filter, and must not be blank
percent	Y*	-	Share of remaining traces to keep, 0 to 100. Used by probabilistic

* = Conditionally required, by type. All of it is checked at load.

Details

Tail sampling decides after the fact, which is the entire point. Head sampling picks traces at the start, before anything is known about them, so an error that happens later is kept only by luck. Buffering the spans and deciding once the trace is complete means the interesting traces can be kept deliberately.

The cost is memory and delay: spans are held for decision_wait_sec before anything is emitted, and the four caps exist to bound what that can consume.

At most one probabilistic policy is allowed, and the configuration is rejected at load if there are two. The source is explicit about why: a single fallback probability is what makes the sampling rate well defined. error, latency and filter are definitive keeps — a trace matching any of them is kept outright — while probabilistic is the fallback applied to what is left, so more than one would make the resulting rate meaningless.

Put the definitive policies first and the probabilistic one last; that ordering is what "keep every error, plus 1% of everything else" looks like.

decision_cache_sec handles the straggler. Spans arriving after their trace was decided — a slow exporter, a long-running span — follow the decision already made rather than starting a new trace or being dropped inconsistently.

Every numeric option treats 0 as "use the default", not "unlimited", and a negative value is rejected at load.

Kept traces go to destination, a named target of the route, the same model Aggregate uses.

Examples

Errors Plus a Sample

Keeping every failing trace, and a small share of the rest...	- tail_sample: decision_wait_sec: 15 destination: traces_target policies: - type: error - type: probabilistic percent: 1
the definitive policy first, the fallback last:	# Every trace containing an error span is kept. # 1% of the remainder is kept for baseline visibility.

Keeping Slow Traces

Adding a latency threshold to the same list...	- tail_sample: destination: traces_target policies: - type: error - type: latency threshold_ms: 2000 - type: probabilistic percent: 5
a trace matching any definitive policy is kept outright:	# Errors and traces of 2s or longer are always kept. # 5% of what neither matched is sampled.

Selecting by Attribute

A filter policy keeps traces matching a condition...	- tail_sample: destination: traces_target policies: - type: filter expression: "service.name == 'checkout'" - type: probabilistic percent: 2
so a service under investigation is retained in full:	# Every checkout trace is kept regardless of outcome, # with 2% of everything else.