Azure IoT Hub

Microsoft Azure Message Queue

Synopsis

Creates an Azure IoT Hub target that publishes records as device-to-cloud messages to Azure IoT Hub, signing in with a device or module connection string, a client certificate or a shared access policy.

Each record goes as one message, over MQTT and always over TLS. The target signs in as an IoT Hub device, or as a module of one. It works like the MQTT target, with the rules of IoT Hub added.

To publish to the MQTT broker of Azure IoT Operations, which runs in your own Kubernetes cluster, use the Azure IoT Operations target instead.

Schema

- name: <string>
  description: <string>
  type: aziothub
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    authentication: <string>
    connection_string: <string>
    endpoint: <string>
    device_id: <string>
    module_id: <string>
    policy_name: <string>
    policy_key: <string>
    transport: <string>
    message_properties: <map>
    batch_size: <numeric>
    max_messages_per_second: <numeric>
    timeout: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    field_format: <string>
    clean_session: <boolean>
    keep_alive: <numeric>
    auto_reconnect: <boolean>
    tls:
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      verify: <boolean>
      ca_name: <string>
      server_name: <string>
      min_tls_version: <string>
      max_tls_version: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be aziothub
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
authentication	N	connection_string	How the target signs in: connection_string, certificate or policy (see Sign-in). In the console: Authentication method (Connection string (SAS), Client certificate (X.509), Shared access policy (SAS))
connection_string	Y, with connection_string	-	The connection string of the device, as the device's page in the Azure portal shows it: HostName=<hub>.azure-devices.net;DeviceId=<device>;SharedAccessKey=<key>. Add ;ModuleId=<module> to sign in as a module. Plain text, an environment variable or a secret. In the console: Connection string
endpoint	Y, with certificate and policy	-	The host name of the IoT hub, such as myhub.azure-devices.net, with no scheme, port or path. With connection_string, the connection string holds the host name. In the console: Address
device_id	Y, with certificate	{director} with policy	With certificate, the device that the client certificate was issued for. With policy, a template, where {director} stands for the name of the director (see One Connection for Each Device). In the console: Device ID or Device ID template
module_id	N	-	With certificate and policy, signs in as this module of the device. With connection_string, add ModuleId to the string instead. In the console: Module ID
policy_name	N	device	With policy, the shared access policy. In the console: Policy name
policy_key	Y, with policy	-	With policy, the primary or the secondary key of that policy. Plain text, an environment variable or a secret. In the console: Policy key
transport	N	mqtt	mqtt connects on port 8883. websocket connects on port 443, for a network that lets only HTTPS through. In the console: Transport (MQTT, port 8883, WebSockets, port 443)
clean_session, keep_alive, auto_reconnect	N	true, 60, true	These set the MQTT connection, as for the MQTT target

The protocol (MQTT 3.1.1), the QoS (1), the retain flag (off), the topic, the client ID, the user name, the password and the last will are fixed by IoT Hub, so the MQTT target's settings for them are not used.

TLS

TLS is always on, so there is no TLS switch.

Field	Required	Default	Description
tls.cert_name	Y, with certificate	-	The client certificate, the device's X.509 identity. With certificate only
tls.key_name	Y, with certificate	-	The certificate's private key. With certificate only
tls.passphrase	N	-	The private key's passphrase. With certificate only
tls.verify	N	true	Checks the IoT hub's certificate
tls.ca_name, tls.server_name, tls.min_tls_version, tls.max_tls_version	N	-	As for the MQTT target. IoT Hub takes TLS 1.2 or newer, so an older tls.min_tls_version counts as 1.2

Data Settings

Field	Required	Default	Description
message_properties	N	-	Application properties that every message carries, as names and text values. A name cannot start with $ or iothub-, which IoT Hub keeps for itself. In the console: Message properties

Performance

Field	Required	Default	Description
batch_size	N	1000	The most records in one batch
max_messages_per_second	N	No limit	The most messages the target publishes in one second, all its workers together. 0 or no value means no limit
timeout	N	30	The seconds the connection, or one publish, may take
max_retries	N	3	How often the target tries a failed message again
retry_delay	N	1	The longest wait before the first retry, in seconds. The wait is random, and its limit doubles with each retry up to 30 seconds
disable_retries	N	false	true makes one attempt for each message, with no retry and no wait. A message that this attempt fails still goes back to the director, which delivers it again, as it does any failed send
field_format	N	-	The format each record is converted to before the target publishes it. See applicable Normalization section

These work as for the MQTT target.

The Console Form

The settings step, Azure IoT Hub settings, has three sections:

• Connection settings: Authentication method and Transport, then the fields of the chosen sign-in.
• Data settings: Message properties, a list of names and values.
• Performance.

The form has no TLS step: the client certificate sits with the Client certificate (X.509) sign-in.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

What Is Sent

Every record, logs and metrics alike, goes as one JSON message.

Where It Goes

Each message goes to the events topic of the device, devices/<device>/messages/events/, or of the module, devices/<device>/modules/<module>/messages/events/. At the end of the topic, a property bag:

• marks the body as JSON in UTF-8, so that IoT Hub message routing can query the body;
• adds device_name, the name of the device that collected the record, when the record has one;
• adds the message properties, in name order.

Sign-in

• Connection string: fits one device. The target makes a Shared Access Signature (SAS) token from the key in the string.
• Client certificate: the target signs in with the device's X.509 certificate and sends no password. Set the device up in the hub for certificate sign-in first.
• Shared access policy: fits several directors. The target makes a SAS token with the key of a policy that has the DeviceConnect permission, such as the hub's built-in device policy. It signs in as the device that device_id names.
• A SAS token is good for 60 minutes. The target makes a new token for every sign-in, and it signs in again every 50 minutes, before the token expires. Messages wait for that short reconnect.

The device has to be registered in the hub before the target signs in as it.

One Connection for Each Device

IoT Hub takes one connection for each device. A second connection with the same device closes the first one.

• All workers of a target share one connection.
• So a target with a connection string or a certificate fits a single director. Two directors that sign in as the same device keep closing each other's connection.
• For several directors, use the shared access policy. With the default device_id of {director}, each director signs in as a device named after it. The target logs the device ID that it signs in as. Register that device in the hub for each director.
• A character that IoT Hub refuses in a device ID becomes a hyphen. When that happens, the target adds a hyphen and eight hex digits made from the device ID before the change, so North West becomes North-West-2878dbaf and stays apart from North-West.

What Is Dropped

• IoT Hub takes a message of 256 KB at the most, the property bag included. The target drops a larger record, counts it as dropped and never tries it again.
• MQTT takes a topic of 65,535 bytes at the most, and the topic holds the property bag. The target drops a record whose topic would be longer in the same way.

When IoT Hub Closes the Connection

IoT Hub closes the connection when it throttles, when the daily message quota of the hub is used up, and when it refuses a message. It does not say which.

• The target logs a warning that names these causes, at most once a minute.
• The client connects again by itself, with auto_reconnect on, the default. The failed messages are then tried again as any failed publish is: by the target, as max_retries and retry_delay say, or, with disable_retries on, by the director, which delivers them again.
• Check the metrics and the quota of the hub in the Azure portal.

How many device-to-cloud sends a second and how many messages a day a hub takes depends on its tier and its units, and each 4 KB of a message counts as one message. See Microsoft's IoT Hub quotas and throttling.

For how a device signs in, see Microsoft's Communicate with an IoT hub using the MQTT protocol and Control access to IoT Hub with Shared Access Signatures.

Examples

One Director with a Device Connection String

One director, the connection string of a device from a secret...

targets: - name: azure_iot_hub type: aziothub properties: connection_string: "$secret{iothub-device-connection-string}"

Several Directors with the Device Policy

Several directors, each signing in as its own device through the hub's device policy...

targets: - name: azure_iot_hub_fleet type: aziothub properties: authentication: policy endpoint: "myhub.azure-devices.net" policy_key: "${IOTHUB_DEVICE_POLICY_KEY}" message_properties: site: "plant-1"

Troubleshooting

Configuration Errors

The director refuses a configuration it cannot use, with one of these messages:

• connection_string is required when authentication is connection_string: copy it from the page of the device in the Azure portal
• connection_string has no HostName part; connection_string has no DeviceId part: use the connection string of a device, not the one of a shared access policy; connection_string has no SharedAccessKey part: to sign in with a certificate, set authentication to certificate
• connection_string is not valid: write each part as Name=value, with a semicolon between the parts
• endpoint is required when authentication is certificate: use the host name of your IoT hub, for example myhub.azure-devices.net, and the same for policy
• endpoint must be a host name alone: leave out the scheme, the port and the path
• device_id is required when authentication is certificate: use the device ID that the client certificate was issued for
• device_id can hold only the placeholder {director}, which stands for the name of the director
• device_id is 130 characters long once {director} is replaced, and IoT Hub takes 128 at most: write a shorter device_id
• policy_key is required when authentication is policy: it is the key of the shared access policy
• policy_key is not valid base64: copy the key again from the Azure portal, and the same for the key in the connection string
• tls.cert_name and tls.key_name are required when authentication is certificate: the client certificate signs the target in to IoT Hub
• message_properties has a property with no name; message_properties.<name> must be text; a name that starts with $ or iothub-
• transport must be "mqtt" or "websocket"; authentication must be "connection_string", "certificate" or "policy"

In the Log While It Runs

• The target connects to Azure IoT Hub as <device>. names the device, or the device and module, that the target signs in as. With a policy, register this device in the hub.
• The connection to Azure IoT Hub was lost: <reason>. IoT Hub closes a connection when it throttles the device, when the daily message quota of the hub is used up, and when it refuses a message. ... comes at most once a minute. See When IoT Hub Closes the Connection.
• In the warning for dropped records: exceeded the 262144 bytes that Azure IoT Hub takes in a message, or had a topic, with its property bag, longer than the 65535 bytes that MQTT takes.

IoT Hub Refuses the Sign-in Again and Again

Cause: the device is not registered in the hub, the key or the certificate is wrong, or another director signs in as the same device.

Fix: register the device that the log names, copy the key or the connection string again from the Azure portal, and give each director its own device.