Sparkplug B

MQTT Industrial IoT

Synopsis

Creates a Sparkplug B target that publishes records to an MQTT broker as one Sparkplug B edge node. Each device that sends records appears as a Sparkplug device. Host applications such as Ignition show the readings of each device as tags.

The target follows Sparkplug 3.0. A host application finds each device and its readings by itself, with nothing to map. Each number or true/false value of a record becomes a metric of its device.

Schema

- name: <string>
  description: <string>
  type: sparkplugb
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    client_id: <string>
    protocol_version: <string>
    timeout: <integer>
    keep_alive: <integer>
    username: <string>
    password: <string>
    group_id: <string>
    edge_node_id: <string>
    primary_host_id: <string>
    device_id: <string>
    device_id_missing_field: <string>
    device_id_fallback: <string>
    device_id_default_value: <string>
    device_timeout: <integer>
    fields: <string[]>
    batch_size: <integer>
    max_messages_per_second: <integer>
    field_format: <string>
    max_retries: <integer>
    retry_delay: <integer>
    disable_retries: <boolean>
    tls:
      status: <boolean>
      verify: <boolean>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      ca_name: <string>
      server_name: <string>
      min_tls_version: <string>
      max_tls_version: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be sparkplugb
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
url	Y	-	The address of the MQTT broker, with its scheme: tcp://, ssl://, ws:// or wss://, such as tcp://broker.example.com:1883. Use ssl:// or wss:// with TLS on. In the console: URL
client_id	N	<group_id>/<edge_node_id>	The MQTT client ID. The default is the same at every start, so the broker knows the node again. In the console: Client ID
protocol_version	N	3.1.1	3.1.1 or 5. In the console: Protocol version
timeout	N	30	The seconds that the connection, and each publish, may take. In the console: Timeout
keep_alive	N	60	How often the target pings the broker while the connection is idle, in seconds. In the console: Keep-alive interval
username, password	N	-	The login for the broker. Plain text, an environment variable or a secret. In the console: Username and Password

Edge Node

Field	Required	Default	Description
group_id	Y	-	The Sparkplug group of the edge node. It is one level of every topic, so it cannot hold +, / or #. In the console: Group ID
edge_node_id	Y	-	The ID of the edge node, which is this director. It must be unique in its group, and the same characters are refused as for group_id. Only one director may run the target (see Before You Start). In the console: Edge node ID
primary_host_id	N	-	The host application that the node follows. With it, the node publishes nothing until that host is online (see How the Primary Host Works). In the console: Primary host ID

Devices

Field	Required	Default	Description
device_id	N	The name of the device that sent the record	The Sparkplug device that each record goes to. A template with record fields in braces, such as {host.name} (see How Device IDs Work). In the console: Device ID
device_id_missing_field	N	drop	What happens to a record that lacks a field the Device ID uses: drop, fallback or default_value. In the console: Missing Device ID field (Drop the record, Use a fallback Device ID, Use a default value), shown once the Device ID holds a field in braces
device_id_fallback	Y, with fallback	-	The device that such a record goes to. It cannot hold a field in braces. In the console: Fallback Device ID
device_id_default_value	N	unknown	The text that takes the place of a missing field, with default_value. In the console: Default value
device_timeout	N	0	The seconds that a device may send no record before the node reports it offline. 0 turns this off (see How the Device Timeout Works). In the console: Device timeout
fields	N	Every number and true/false value	The fields to send. An entry also takes every field inside it (see How Metrics Work). In the console: Fields to send

Delivery Settings

Field	Required	Default	Description
batch_size	N	1000	The most records in one batch. In the console: Batch size
max_messages_per_second	N	No limit	The most messages the target publishes in one second. Every message counts, births and deaths included. 0 or no value means no limit. In the console: Maximum messages per second
field_format	N	-	The format each record is converted to before the target publishes it. See applicable Normalization section. In the console: Field format
max_retries	N	3	How often the target tries a failed message again. In the console: Maximum retries
retry_delay	N	1	The longest wait before the first retry, in seconds. In the console: Retry delay
disable_retries	N	false	true makes one attempt for each message, with no retry and no wait. In the console: Disable retries

TLS

Field	Required	Default	Description
tls.status	N	false	Turns on the TLS settings below. The scheme in url decides whether the connection uses TLS, so use ssl:// or wss:// with it. In the console: TLS encryption
tls.verify	N	true	Checks the broker's certificate. In the console: Certificate verification, with TLS on
tls.cert_name, tls.key_name	N	-	A client certificate and its private key, for a broker that asks for one. Give both or neither. In the console: Certificate and Private key, with TLS on
tls.min_tls_version, tls.max_tls_version	N	-	As for the MQTT target. In the console: Minimum TLS version and Maximum TLS version
tls.passphrase, tls.ca_name, tls.server_name	N	-	As for the MQTT target. Not in the console.

Settings That Sparkplug Fixes

Sparkplug fixes some of the MQTT settings. Births and data go out at QoS 0 and are never retained, the session is always clean, and the target always reconnects. So the MQTT target's qos, retained, clean_session, auto_reconnect, topic and will have no effect here. The target gives each connection a last will of its own (see What the Host Sees).

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

Before You Start

• A Sparkplug host application, such as Ignition, must read from the same broker. It shows what the target publishes.
• Run the target on one director only. A target runs on every director unless it has a director restriction, so turn on Director restriction in General settings and pick one director. Two directors with the same edge node cause trouble. With the default client ID, each one takes the other's connection, so the host sees the node go offline and online again and again. With client IDs of their own, both stay connected, and the host takes their messages as those of one node.
• For a second director, add a second target with another edge_node_id.

What the Host Sees

• When the target connects, the node publishes its birth (NBIRTH, the message that says the node is online). With a primary host, it waits until that host is online first. Then it publishes a birth (DBIRTH) for each device it knows, with every metric of that device and its value. A host builds its list of tags from the births.
• Each connection has a last will: the node's death (NDEATH, the message that says the node is offline). When the connection ends without a goodbye, the broker publishes it, and the host marks the node and its devices offline.
• When the director stops the target, the node tells the host that it is gone.
• A host can ask the node to publish its births again (the Rebirth command). The node ignores every other command, because the target sends readings only. It never writes a value that a host sends.

Only Changed Values Are Sent

• A device is born with the metrics of its first record.
• After that, the node publishes a metric only when its value changes. It sends the changed metrics in a data message (DDATA). A record that changes no value publishes nothing, and it counts as delivered.
• A record that brings a metric that the device did not have makes the device die (DDEATH, the message that says a device is offline) and be born again (DBIRTH), with every metric. A host keeps a fixed list of metrics for each device, so this is how a new metric reaches it.
• A host keeps the type of a metric from its birth. When a record brings a value of another type for that metric, the target leaves the value out.

How Device IDs Work

• Without device_id, each record goes to the Sparkplug device that has the name of the device that sent the record. A record from a device with no name is dropped.
• With device_id, each {field.path} in the template takes the value of that field in the record. A Device ID is one level of a topic, so each +, / and # in it becomes _.
• A value is missing when the record does not have the field, or when the field holds null, an empty text, an object or a list. Then device_id_missing_field decides: drop drops the record, fallback sends it to device_id_fallback, and default_value puts device_id_default_value where the value would go.

How Metrics Work

• Without fields, each number of a record becomes a metric of type Double, and each true/false value a metric of type Boolean. Text, lists and nulls are not sent, and neither is @timestamp.
• The name of a metric is the path to its value, with / between the levels: the field modbus.register.holding becomes the metric modbus/register/holding.
• Each metric carries the time of its record.
• fields limits what is sent. An entry is the path to a field, with a dot between the names, such as cpu.user. When the field is an object, every number, true/false value and text inside it is sent. With fields, a text value is sent too, as a metric of type String.
• Every number is sent as a Double, whole numbers included.
• A record that has no Device ID, or no metric to send, is dropped.

How the Primary Host Works

• Without primary_host_id, the node publishes as soon as it connects.
• With primary_host_id, the node publishes nothing until that host says that it is online. The host says so in a STATE message, which it keeps on the broker. Until then, each batch fails, and the director keeps the records and sends them again later.
• When the host goes offline, the node ends its connection, connects again and waits for the host. Records wait in the director meanwhile.
• The host must follow Sparkplug 3.0. A host that follows an older version announces itself in another way, which the node does not read, so the node would wait forever. For such a host, leave primary_host_id empty.

How the Device Timeout Works

• With device_timeout above 0, a device that sends no record for that many seconds is reported offline (DDEATH).
• The next record of the device gives it a new birth, with every metric the node knows for it.
• With 0, a device is never reported offline because of time.

Sending Again

• When a message fails, the director sends the whole batch again. The records that went out before publish nothing then, because the node sends a metric only when its value changed, and never a value older than the one it has.
• A publish that timed out can still have reached the broker, so such a value can go out twice.

Dropped Records

A dropped record counts as dropped in the target's statistics, and the rest of the batch is still sent. With debug.status on, the log says how many records were dropped and why.

Examples

Basic

The simplest form: each record goes to the device it came from:

targets:
  - name: sparkplug
    type: sparkplugb
    properties:
      url: "tcp://broker.example.com:1883"
      group_id: "plant-1"
      edge_node_id: "director-1"

Device IDs from a Record Field, over TLS

Device IDs from a record field, a few fields, a primary host and a device timeout, over TLS:

targets:
  - name: sparkplug_modbus
    type: sparkplugb
    properties:
      url: "ssl://broker.example.com:8883"
      group_id: "plant-1"
      edge_node_id: "director-1"
      primary_host_id: "scada-host"
      device_id: "{host.name}"
      device_id_missing_field: default_value
      fields:
        - modbus.register.holding
        - modbus.register.input
      device_timeout: 300
      username: "director-1"
      password: "$secret{id=1001}"
      tls:
        status: true

MQTT 5 with a Rate Limit

targets:
  - name: sparkplug_v5
    type: sparkplugb
    properties:
      url: "tcp://10.0.0.20:1883"
      group_id: "line-3"
      edge_node_id: "director-2"
      protocol_version: "5"
      max_messages_per_second: 200

Troubleshooting

Configuration the target refuses

Cause: the director refuses a configuration it cannot use. The message names the setting:

• url is required for sparkplugb target
• group_id is required for sparkplugb target, and edge_node_id is required for sparkplugb target
• group_id must not contain "+", "/", "#" or a NUL character, and must be valid UTF-8 text, and the same for edge_node_id and primary_host_id
• protocol_version must be "3.1.1" or "5"
• max_messages_per_second must be 0 or more, and device_timeout must be 0 or more
• both tls.cert_name and tls.key_name must be provided together or omitted together
• device_id_fallback is required when device_id_missing_field is "fallback", device_id_fallback "…" is used as it is, so it cannot hold a placeholder, and device_id_missing_field must be "drop", "fallback" or "default_value", not "…"
• device_id is not valid: …, followed by what is wrong with the template: a brace without its partner, an empty placeholder {}, or a NUL character.

Fix: correct the setting the message names.

"the target waits for the primary host … to be online"

the target waits for the primary host "scada-host" to be online

Cause: the host has not said that it is online.

Fix: check that the host runs, that it follows Sparkplug 3.0, and that its host ID is the same as primary_host_id. The director keeps the records meanwhile.

"the primary host … is offline, so the target is connecting again"

the primary host "scada-host" is offline, so the target is connecting again

Cause: the host went offline, and the node waits for it on a new connection.

Fix: bring the host back online. The records wait in the director.

"the edge node is not ready"

the edge node is not ready: it has not published its birth certificate on this connection yet

Cause: a batch came while the node was connecting.

Fix: none is needed. The director sends the batch again.

The node goes offline and online again and again

Cause: two directors run the target with the same IDs.

Fix: give the target a director restriction, so that one director runs it (see Before You Start).

Messages in the debug log

With debug.status on, the log also shows:

• Dropped N of M records in this batch:, with had no device ID or had no metric.
• The device "…" has the metric "…" with another datatype, so the target leaves this value out.
• The edge node reported N devices offline, as they sent no record for …
• The primary host "…" is online. and The primary host "…" is offline.
• The target ignores a STATE message that it cannot read: …: a message on the primary host's STATE topic is not a Sparkplug 3.0 STATE message.