Set Math

Mutate

Synopsis

Applies a set operation — union, difference or intersect — across two or more array fields, and writes the resulting array to a target field.

Schema

- set_math:
    operation: <enum>
    fields: <string[]>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    ignore_missing: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>
    target_field: <ident>

Configuration

The following fields are used to define the processor:

Field	Required	Default	Description
operation	Y	-	union, difference or intersect. Matched case-insensitively; any other value is an error
fields	Y	-	Array fields to operate on, in order. At least two are required
description	N	-	Explanatory note
if	N	-	Condition to run
ignore_failure	N	false	See Handling Failures
ignore_missing	N	false	If true, quietly exit if field doesn't exist
on_failure	N	-	See Handling Failures
on_success	N	-	See Handling Success
tag	N	-	Identifier
disabled	N	false	When true, the processor is skipped and the event continues to the next one. Lets you take a processor out of the path without removing its configuration
target_field	N	first entry of fields	Field to store the result. Defaults to the first field listed, replacing it

Details

Operation	Result
union	Every distinct element from all the fields
difference	Elements of the first field that appear in none of the others
intersect	Elements of the first field that appear in all of the others

Two of the three are asymmetric: difference and intersect both start from the first field and treat the rest as the comparison set, so field order changes the answer. Only union is order-independent, and even there the order of the output follows the order the elements were first seen.

Duplicates are removed from the result in every operation. The output is a set, so an element appearing twice in an input appears once in the output.

A field that holds something other than an array is treated as a single-element set rather than failing, so a scalar can take part in a union without being wrapped first.

Missing fields follow the same rule as Object Merge: an individual missing field is skipped, and only when every field is missing is it an error. The result is always an array — an operation that eliminates everything writes an empty array rather than null.

Examples

Union

Combining tag lists, dropping duplicates...	{ "rule_tags": ["persistence", "t1053"], "analyst_tags": ["t1053", "reviewed"] } - set_math: operation: union fields: ["rule_tags", "analyst_tags"] target_field: all_tags
each element appearing once:	{ "rule_tags": ["persistence", "t1053"], "analyst_tags": ["t1053", "reviewed"], "all_tags": ["persistence", "t1053", "reviewed"] }

Difference

Finding the ports that are open but not expected...	{ "observed_ports": [22, 80, 443, 8080], "allowed_ports": [22, 443] } - set_math: operation: difference fields: ["observed_ports", "allowed_ports"] target_field: unexpected_ports
keeping only what the first field has and the second does not:	{ "observed_ports": [22, 80, 443, 8080], "allowed_ports": [22, 443], "unexpected_ports": [80, 8080] }

Intersect

Keeping only what every source agrees on...	{ "feed_a": ["1.2.3.4", "5.6.7.8", "9.9.9.9"], "feed_b": ["5.6.7.8", "9.9.9.9"], "feed_c": ["9.9.9.9", "1.1.1.1"] } - set_math: operation: intersect fields: ["feed_a", "feed_b", "feed_c"] target_field: corroborated
an address must appear in all three to survive:	{ "feed_a": ["1.2.3.4", "5.6.7.8", "9.9.9.9"], "feed_b": ["5.6.7.8", "9.9.9.9"], "feed_c": ["9.9.9.9", "1.1.1.1"], "corroborated": ["9.9.9.9"] }

No Overlap

An operation that eliminates everything still writes an array...	{ "observed_ports": [22, 443], "allowed_ports": [22, 443] } - set_math: operation: difference fields: ["observed_ports", "allowed_ports"] target_field: unexpected_ports
an empty one rather than null:	{ "observed_ports": [22, 443], "allowed_ports": [22, 443], "unexpected_ports": [] }