Event Split Rules

Synopsis

Event split rules split one incoming message into separate logs. For example, an Azure Event Hubs message {"records":[...]} that holds 35 records becomes 35 logs. From that point on, every count shows 35 logs, including the device's own In count.

Azure sends Diagnostic Settings, Activity Log, Entra ID and Defender XDR data to Event Hubs in batches: each message holds many logs. Without rules, the device treats each message as one log, unless its split_mode splits it. In one Defender XDR capture, 13 messages held 259 logs: the console counted 13, and the Azure packs, which expect one log per record, could not map them. Event split rules make each record a log of its own before anything else reads it, so pipelines do not have to split them.

A device names the rule sets it uses in its event_split_rules property. These device types take it. They are the ones that receive whole messages:

Device	Type
Event Hubs	eventhubs, in Kafka endpoint mode too
Azure Service Bus	servicebus
Kafka	kafka
Google Cloud Pub/Sub	pubsub
HTTP	http

Configuration

event_split_rules is an ordered list of rule set keys:

properties:
  event_split_rules:
    - azure_records
    - json_lines

Each rule set is a list of rules. The Director tries the rules in order, across the rule sets in the order the device lists them, and the first rule that matches splits the message. When no rule matches, the device handles the message as it would without rules: it follows its split_mode, or keeps the message whole.

The Director splits each message as the device receives it, before anything else reads it. Event Hubs, Service Bus, Kafka and Pub/Sub still acknowledge a message only after all of its logs are delivered.

Where Rule Sets Come From

The platform's Library holds rule sets, and sends each Director the rule sets that its devices list in event_split_rules. The Director has no built-in rule sets of its own. Built-in Library rule sets, such as azure_records, reach it in the same way as any other rule set.

A standalone Director reads each rule set from config/library/event_split_rule/<key>.json, in the format the Library stores:

{
  "name": "Azure records",
  "inspectBytes": 256,
  "rules": [
    {
      "name": "Records",
      "enabled": true,
      "condition": "message =~ /^\\s*\\{\\s*\"records\"\\s*:\\s*\\[/",
      "type": "json_array",
      "settings": { "arrayField": "records" },
      "maxLogSize": 10485760
    }
  ]
}

Rule Sets

Field	Required	Default	Description
name	N	-	Rule set name
inspectBytes	N	256	How much of each message, from its start, a condition can read: 16 to 4096 bytes
rules	Y	-	1 to 50 rules. Each rule needs a name of its own

Rules

Field	Required	Default	Description
name	Y	-	Rule name, unique within the rule set
enabled	N	false	Only a rule with enabled: true is tried. A rule that is turned off is skipped
condition	N	-	Which messages the rule applies to (see Conditions). An empty condition matches every message
type	Y	-	How the rule splits a message (see Rule Types)
settings	N	-	The type's settings
maxLogSize	N	10485760 (10 MB)	The largest log the rule makes, from 1024 bytes to 10 MB. The Director cuts a longer log into pieces, so no data is lost, but a JSON log that was cut is no longer valid JSON

Conditions

A condition is written in the pipeline expression language, as a processor's if: is. It can read:

• message: the first inspectBytes bytes of the message
• source: the place the message came from (below)
• device.type and device.name

Device type	source
eventhubs	The event hub, in Kafka endpoint mode too
kafka	The topic
servicebus	The queue, or <topic>/<subscription>
pubsub	The subscription ID
http	The path the device listens on (url)

Rule Types

Type	What it does	Settings
none	Keeps the whole message as one log	-
lines	Makes each line a log. A line that starts with a space or a tab stays with the line before it, as a stack trace's lines do. A blank line ends a log	-
separator	Splits at each occurrence of a fixed string. Blank pieces are dropped	separator
regex	Splits where a pattern matches. Each match marks the start or the end of a log. The pattern can use lookahead and lookbehind, and must not match an empty string	pattern, boundary (start or end)
jsonl	Makes each JSON value a log: JSON lines, or JSON objects written over several lines	-
json_array	Makes each element of a JSON array a log. The array is the whole message, or the field arrayField names, such as records or data.items	arrayField, copyFields, removeFields

A json_array rule can change the logs it makes:

• copyFields lists fields of the message to add to each log, under the field's last name. For example, meta.region is added as region. A log that already has a field of that name keeps its own. Across all of a message's logs, the copied fields can add at most the size of the message, or 1 MB for a message smaller than 1 MB. If the copied fields would add more, the Director does not split the message and keeps it as one log.
• removeFields lists fields to remove from each log, such as properties.raw.

A log keeps the bytes it arrived with, unless its rule copies or removes fields.

Details

What Happens to a Message

Situation	Result
A rule matches and splits the message	Each piece is a log of its own, from the start. Every later count counts the logs
No rule matches	The device behaves as it does without rules. It follows its split_mode, or keeps the message whole
A rule matches but cannot split the message, for example a json_array rule given text that is not JSON	The message is kept whole, as one log. The Director logs a warning at most once a minute. Later rules are not tried
A log is longer than the rule's maxLogSize	The log is cut into pieces. No bytes are lost, but a cut JSON log is no longer valid JSON
A message would make too many logs	The message is kept whole, as one log (see Limits)
A message's logs are too large to travel together	The message is kept whole, as one log. All of a message's logs travel in one payload, so that its source acknowledges it only once they are all delivered. When the device's buffer, or one payload, cannot hold them all, the message is not split
The device names a rule set the Director does not have, or one that is not valid	The Director skips that rule set and logs a warning at most once a minute

Limits

A json_array rule splits an array of at most 10,000 elements. Every rule makes at most 10,000 logs of a message, or one log for every 16 bytes of a larger message. This count includes the pieces of logs that were cut because they were too long. When a rule would make more, the Director keeps the message whole, as one log.

The same limit applies when the device's split_mode splits a message that no rule matched.

Warnings

The Director writes a warning to its log at most once a minute for each of these:

• It does not have a rule set the device names. It skips that rule set.
• A rule set is not valid. It skips that rule set.
• A rule matched a message but could not split it, for example a json_array rule given text that is not JSON, or a message that would make too many logs. It keeps the message whole, as one log.
• A message's logs are too large to be handed on together. It keeps the message whole, as one log.

Direct Path

A device with event split rules never takes the direct path, because the direct path sends each message whole.

Edits and Restarts

A rule set that you edit applies from the next message, without restarting the device. Changing the device's event_split_rules list, including its order, restarts the device (see Changes That Restart a Device).

Crash Recovery

With crash_resistance, a device's queue files on disk are recovered after a crash as they were written:

• A file written while the device had rules holds logs that were split already, so they are not split again.
• A file written before the device had rules holds whole messages. They are split by the device's split_mode, as they would have been before the crash, and not by the rules.