AWS IoT Core

Amazon AWS Message Queue

Synopsis

Creates an AWS IoT Core target that publishes records as messages to AWS IoT Core topics, over MQTT with a device certificate or over HTTPS with AWS credentials. Supports topics taken from record fields, Basic Ingest, MQTT 5 and a last will.

Schema

- name: <string>
  description: <string>
  type: awsiotcore
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    endpoint: <string>
    authentication: <string>
    topic: <string>
    topic_missing_field: <string>
    topic_fallback: <string>
    topic_default_value: <string>
    basic_ingest_rule: <string>
    qos: <integer>
    retained: <boolean>
    port: <integer>
    client_id: <string>
    protocol_version: <string>
    clean_session: <boolean>
    keep_alive: <integer>
    auto_reconnect: <boolean>
    tls:
      verify: <boolean>
      ca_name: <string>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      server_name: <string>
      min_tls_version: <string>
      max_tls_version: <string>
    will:
      status: <boolean>
      topic: <string>
      payload: <string>
      qos: <integer>
      retained: <boolean>
    key: <string>
    secret: <string>
    session: <string>
    region: <string>
    batch_size: <integer>
    timeout: <integer>
    max_messages_per_second: <integer>
    max_retries: <integer>
    retry_delay: <integer>
    disable_retries: <boolean>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be awsiotcore
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
endpoint	Y	-	The host name of the data endpoint of the AWS account and Region, which aws iot describe-endpoint --endpoint-type iot:Data-ATS prints, such as a1b2c3d4e5f6g7-ats.iot.eu-west-1.amazonaws.com. The host name alone: no scheme, port or path. Plain text or an environment variable. In the console: Endpoint
authentication	N	certificate	How the target signs in: certificate, a device certificate over MQTT, or iam, AWS credentials over HTTPS. In the console: Authentication method (Device certificate (MQTT), AWS credentials (HTTPS))

Device Certificate (MQTT)

These fields apply with authentication: certificate only.

Field	Required	Default	Description
port	N	8883	8883, or 443 for a network that allows HTTPS only. On 443 the target asks AWS IoT Core for MQTT with ALPN. In the console: Port
client_id	N	Generated	The MQTT client ID. AWS IoT Core allows one connection for each client ID, so use the ID the device's IoT policy allows, often the thing name. In the console: Client ID
protocol_version	N	3.1.1	3.1.1 or 5. In the console: Protocol version
clean_session	N	true	In the console: Clean session
keep_alive	N	60	How often the target pings AWS IoT Core while the connection is idle, in seconds. In the console: Keep-alive interval
auto_reconnect	N	true	Reconnects after a lost connection. Not in the console.
tls.cert_name	Y	-	The device certificate. It must be registered and active in AWS IoT Core. Its content, a file path, an environment variable or a secret. In the console: Certificate
tls.key_name	Y	-	The certificate's private key, in the same forms. In the console: Private key
tls.passphrase	N	-	The private key's passphrase. In the console: Passphrase
tls.ca_name	N	-	The CA that signed the AWS IoT Core certificate. Empty trusts Amazon Root CA 1. In the console: CA certificate
tls.verify	N	true	Checks the AWS IoT Core certificate. In the console: Certificate verification
tls.server_name, tls.min_tls_version, tls.max_tls_version	N	-	As for the MQTT target. Not in the console.

TLS is always on with a device certificate, so there is no TLS switch.

AWS Credentials (HTTPS)

These fields apply with authentication: iam only.

Field	Required	Default	Description
key	N	-	The access key ID. Plain text, an environment variable or a secret. Without key and secret, the director uses its own AWS credentials: from its environment, a shared profile or the role of its host. In the console: Access key ID
secret	N	-	The secret access key. The keys are used only when both are set. In the console: Secret access key
session	N	-	The session token of temporary credentials. In the console: Session token
region	N	The endpoint's Region	The Region each request is signed for. Required when the endpoint does not name its Region, as a custom domain does not. In the console: Region

Topic

Field	Required	Default	Description
topic	Y	-	The topic of each message. It can take part of its name from each record, as in plant/{site}/telemetry (see Topic per Record below). At most 256 bytes and 7 slashes, not counting the fields it names. In the console: Topic
topic_missing_field	N	drop	What happens to a record that lacks a field the topic names: drop, fallback or default_value. In the console: the Missing topic field choice (Drop the record, Send it to a fallback topic, Use a default value)
topic_fallback	N	-	The topic for those records. Required when topic_missing_field is fallback. It is used as written, so it cannot hold braces, +, # or a NUL character, and it has the same limits as topic. In the console: Fallback topic
topic_default_value	N	unknown	The text that takes the place of a missing value when topic_missing_field is default_value. It cannot hold /, +, # or a NUL character. In the console: Default value
basic_ingest_rule	N	-	The name of an AWS IoT rule: letters, digits and underscores, up to 128. Each message then goes to the topic $aws/rules/<rule>/<topic>, straight to that rule (AWS calls this Basic Ingest). The prefix does not count toward the topic limits. In the console: Basic Ingest rule
qos	N	1	0 or 1; AWS IoT Core has no QoS 2. In the console: Quality of service
retained	N	false	Publishes each message as a retained message. AWS IoT Core takes at most one retained message a second for each topic, so set it only for topics that change more slowly than that. In the console: Retained messages

AWS IoT Core Settings

Field	Required	Default	Description
batch_size	N	1000	The most records in one batch. In the console: Batch size
timeout	N	30	The seconds the connection, or one publish, may take. In the console: Timeout
field_format	N	-	The format each record is converted to before the target publishes it, such as ecs or ocsf. See applicable Normalization section. In the console: Field format
max_messages_per_second	N	100 with a device certificate; no limit with AWS credentials	The most messages the target publishes in one second, all its workers together. With a device certificate, 100 is also the highest value, because AWS IoT Core discards what one connection publishes beyond 100 a second. 0 means the default. In the console: Maximum messages per second
max_retries	N	3	How often the target tries a failed message again. In the console: Maximum retries
retry_delay	N	1	The longest wait before the first retry, in seconds. The wait is random, and its limit doubles with each retry up to 30 seconds. In the console: Retry delay
disable_retries	N	false	true makes one attempt for each message, with no retry and no wait. In the console: Disable retries

Last Will

These fields sit in the will block, and apply with a device certificate only.

Field	Required	Default	Description
will.status	N	false	AWS IoT Core publishes the last will when the director drops off without closing its connection. In the console: the Last will switch
will.topic	Y, when the last will is on	-	The topic of the last will. It cannot hold +, # or a NUL character, and it has the same limits as topic. In the console: Last will topic
will.payload	N	-	The text of the last will. In the console: Last will payload
will.qos	N	0	0 or 1. In the console: Last will QoS
will.retained	N	false	In the console: Retained last will

A last will with AWS credentials is a configuration error: a last will needs MQTT.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

Topic per Record

• The topic can hold record fields in braces, such as plant/{site}/telemetry. The target puts the value of each field in its place, and the message goes to the topic that this gives.
• A value cannot add a level to the topic: its /, +, # and NUL characters become _. It can only make the topic longer.
• A field is missing when it is absent, or holds null, an empty text, an object or an array. topic_missing_field then decides what happens.

AWS IoT Core Limits

• The target drops a record that AWS IoT Core would refuse every time, and counts it as dropped: a message over 128 KB (131,072 bytes), a topic over 256 bytes, or a topic with more than 7 slashes. The batch still succeeds. With debug.status on, the log names the reason.
• The limits of the topic do not count the Basic Ingest prefix.
• A topic whose own text, without the fields it names, is already over a limit is a configuration error, because every record would be dropped. So is such a fallback topic or last will topic.
• AWS IoT Core takes at most 100 messages a second from one MQTT connection, and discards the rest without an error. With a device certificate the target keeps to that rate, and records wait their turn. For a higher rate, sign in with AWS credentials.

Device Certificate (MQTT)

• All workers of the target share one MQTT connection.
• On port 443, the target names the ALPN protocol x-amzn-mqtt-ca, which AWS IoT Core needs on that port.
• The IoT policy attached to the certificate must allow iot:Connect for the client ID and iot:Publish for the topics. With Basic Ingest, the topic AWS checks is $aws/rules/<rule>/<topic>.
• With retained or will.retained on, the policy must also allow iot:RetainPublish for those topics.
• With the last will on, a policy that limits iot:Connect with the iot:ConnectAttributes condition must allow LastWill.

AWS Credentials (HTTPS)

• Each message is one HTTPS request to the endpoint, signed with AWS Signature Version 4.
• The IAM identity must be allowed iot:Publish on the topics, and iot:RetainPublish as well with retained on.
• The status of AWS's answer decides what happens to a message:
  • 400 or 413: AWS refuses this message every time. The target drops it, counts it as dropped, and the batch goes on.
  • 401 or 403: the sign-in or the permission is wrong. The target drops no record. It retries the message as max_retries and retry_delay say. If every try fails, the batch fails, and the director sends it again.
  • 429 or 5xx: the message may go through later. The target retries it in the same way.

Examples

Device Certificate

Publishing over MQTT with a device certificate:

targets:
  - name: iot_telemetry
    type: awsiotcore
    properties:
      endpoint: "a1b2c3d4e5f6g7-ats.iot.eu-west-1.amazonaws.com"
      topic: "plant/telemetry"
      tls:
        cert_name: "$path{certs/device.pem.crt}"
        key_name: "$path{certs/device.pem.key}"

Topic per Site with Basic Ingest

A topic per site, sent to an AWS IoT rule through Basic Ingest:

targets:
  - name: iot_per_site
    type: awsiotcore
    properties:
      endpoint: "a1b2c3d4e5f6g7-ats.iot.eu-west-1.amazonaws.com"
      topic: "plant/{site}/telemetry"
      topic_missing_field: fallback
      topic_fallback: "plant/unknown/telemetry"
      basic_ingest_rule: "forward_telemetry"
      tls:
        cert_name: "$path{certs/device.pem.crt}"
        key_name: "$path{certs/device.pem.key}"

AWS Credentials with the Director's Role

Publishing over HTTPS with the director's own AWS role:

targets:
  - name: iot_https
    type: awsiotcore
    properties:
      endpoint: "a1b2c3d4e5f6g7-ats.iot.eu-west-1.amazonaws.com"
      authentication: iam
      topic: "plant/telemetry"

AWS Credentials in Secrets

Publishing over HTTPS with AWS credentials kept in secrets:

targets:
  - name: iot_https_keys
    type: awsiotcore
    properties:
      endpoint: "a1b2c3d4e5f6g7-ats.iot.eu-west-1.amazonaws.com"
      authentication: iam
      key: "$secret{iot-access-key}"
      secret: "$secret{iot-secret-key}"
      topic: "plant/telemetry"

Troubleshooting

Configuration the target refuses

Cause: the director refuses a configuration it cannot use. The message names the setting:

• endpoint is required for awsiotcore target: use the host name of your AWS IoT Core endpoint, for example xxxxxxxxxxxxxx-ats.iot.eu-west-1.amazonaws.com, and endpoint must be a host name alone: leave out the scheme, the port and the path
• authentication must be "certificate" or "iam"
• qos must be 0 or 1: AWS IoT Core does not take QoS 2, and will.qos must be 0 or 1: AWS IoT Core does not take QoS 2
• basic_ingest_rule must be the name of a rule: letters, digits and underscores, up to 128 characters
• port must be 8883 or 443
• tls.cert_name and tls.key_name are required when authentication is certificate: the client certificate signs the target in to AWS IoT Core
• max_messages_per_second must be 0 or more, and max_messages_per_second must be 100 or less when authentication is certificate: AWS IoT Core discards what a connection publishes above that rate
• topic must be at most 256 bytes long, with at most 7 slashes, not counting the fields it names: AWS IoT Core refuses a longer topic, and the same for topic_fallback and will.topic
• will.topic is required when will.status is on, and will.topic must not contain the wildcard characters + or #, or a NUL character
• will.status is on, but the last will needs MQTT: use authentication certificate
• region is required when authentication is iam: the endpoint does not name its region, so set it, for example eu-west-1
• The checks of the topic template itself: a brace without its partner, an empty placeholder {}, a +, # or NUL character outside a placeholder, a topic_missing_field other than drop, fallback or default_value, a missing topic_fallback under fallback, and a topic_fallback or topic_default_value that holds a character it cannot hold.

Fix: correct the setting the message names.

Records dropped

With debug.status on, the log says how many records of a batch were dropped, and why:

• … exceeded the 131072 bytes that AWS IoT Core takes in a message
• … had a topic longer than the 256 bytes that AWS IoT Core takes
• … had a topic with more than the 7 slashes that AWS IoT Core takes
• AWS IoT Core refused …

Cause: the records broke a limit of AWS IoT Core, or, with AWS credentials, AWS refused them for what is in them (see AWS IoT Core Limits above).

Fix: reduce the record size in a pipeline, or shorten the topic.

Errors over HTTPS

AWS IoT Core refused the message (HTTP <status>): <answer>
AWS IoT Core refused the sign-in or the permission (HTTP 401|403): <answer>
AWS IoT Core answered HTTP <status>: <answer>

Cause:

• refused the message means AWS refused that message for what is in it. It is dropped.
• refused the sign-in or the permission means the keys, the role or the IAM policy is wrong.
• answered HTTP <status> means AWS throttled the request or failed. The message is retried.

Fix: for a sign-in or permission error, check key, secret and session, the role of the director's host, and that the IAM policy allows iot:Publish on the topics.

No AWS credentials

no AWS credentials to sign the requests with: set key and secret, or give the host an AWS role
could not get the AWS credentials to sign the message with: …

Cause: the director found no AWS credentials.

Fix: set key and secret, or give the director's host an AWS role.

"failed to load TLS certificate"

failed to load TLS certificate: …

Cause: the device certificate or its key could not be read.

Fix: check tls.cert_name, tls.key_name and, for an encrypted key, tls.passphrase.