Schema Drift

Analytics State Management

Synopsis

Learns each source's field set, then reports when it changes — a new field, a removed field, or a field whose type has changed.

Schema

- schema_drift:
    target_field: <ident>
    learn_records: <numeric>
    cooldown_sec: <numeric>
    absence_sec: <numeric>
    max_fields: <numeric>
    allowlist: <string[]>
    on_drift: <processor[]>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration

No field is required — the processor runs with its defaults and learns whatever arrives.

Field	Required	Default	Description
target_field	N	schema_drift	Field that receives the diff payload, on the record that fires an alert
learn_records	N	100	Records observed per source before the baseline seals and deviations start alerting
cooldown_sec	N	300	Minimum seconds between alerts per source. Changes detected inside the window batch into the next alert
absence_sec	N	900	A baseline field unseen for this long is declared removed
max_fields	N	1000	Cap on the per-source fingerprint size
allowlist	N	-	Top-level field names excluded from the fingerprint entirely. Exact names, not patterns
on_drift	N	-	Processors to run only on the record that fired an alert, after target_field is written
description	N	-	Explanatory note
if	N	-	Condition to run
ignore_failure	N	false	See Handling Failures
on_failure	N	-	See Handling Failures
on_success	N	-	See Handling Success
tag	N	-	Identifier

Details

The processor keeps a fingerprint per source — the set of top-level field names and their coarse types. For the first learn_records records from a source it only learns; after that the baseline is sealed and any deviation is a candidate alert.

Every numeric option treats 0 as "use the default", not as "disable". cooldown_sec: 0 gives the 300-second default rather than alerting on every detection, and absence_sec: 0 gives 900 seconds rather than disabling removal detection. The source comments are explicit that this follows aggregate's window_sec precedent, on the grounds that a storm-free default is the safe one for a notification path. A negative value is a configuration error, rejected at load.

on_drift runs on the alerting record only, after the payload has been written — it is the notification hook, for a Slack, Teams or PagerDuty step. Ordinary on_success runs on every record, which is not what a drift notification wants.

The configuration is validated at load, not per record, and the processor's state is keyed on a hash of every option. Editing any option therefore starts fresh baselines, while an untouched processor re-attaches to its existing baselines across a config reload.

The payload

The diff written to target_field always carries:

Key	Contents
source	The source label the baseline is kept under
device_id, device_type, definition_id	Which input produced the record
baseline_fields	Size of the sealed baseline
detected_at	RFC 3339 timestamp of the detection

And carries these only when non-empty, so their presence is itself the signal of what kind of drift occurred:

• new_fields — names not in the baseline
• removed_fields — baseline names unseen for absence_sec
• type_changes — names whose coarse type changed

Field names are sorted, so the payload is stable between runs.

Examples

Watching a Source

Learning the field set, then alerting on changes...	- schema_drift: learn_records: 500 target_field: drift
the payload appears on the triggering record only:	{ "message": "auth success", "drift": { "source": "device:42", "device_id": "42", "device_type": "syslog", "definition_id": "auth", "baseline_fields": 18, "detected_at": "2026-09-02T14:37:12Z", "new_fields": ["mfa_method"] } }

Excluding Volatile Fields

Fields expected to come and go would otherwise alert constantly...	- schema_drift: allowlist: - trace_id - request_id - session_token
so they are left out of the fingerprint entirely:	# A record carrying trace_id no longer counts as new-field drift, # and its absence is not removal drift either.

Notifying on Drift

on_drift runs only on the record that fired the alert...	- schema_drift: target_field: drift cooldown_sec: 3600 on_drift: - slack: webhook: "$secret{id=12}" message: "Schema drift on {{drift.source}}: {{drift.new_fields}}"
with an hour between alerts per source, so a change storm sends one message:	# Changes detected during the cooldown are batched # into the next alert rather than each sending a message.