Amazon Timestream for InfluxDB

Amazon AWS Databases

Synopsis

Creates an Amazon Timestream for InfluxDB target that writes each record as one point of InfluxDB line protocol to a Timestream for InfluxDB instance, the InfluxDB that AWS runs as a managed service. It writes to InfluxDB 2 through the v2 write API, or to InfluxDB 3 through the v3 write API, always over HTTPS.

Schema

- name: <string>
  description: <string>
  type: amazontimestream
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <integer>
    api_version: <string>
    org: <string>
    bucket: <string>
    database: <string>
    token: <string>
    measurement_field: <string>
    measurement: <string>
    tags: <string[]>
    fields: <string[]>
    batch_size: <integer>
    max_bytes: <integer>
    timeout: <integer>
    field_format: <string>
    tls:
      verify: <boolean>
      ca_name: <string>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      server_name: <string>
      min_tls_version: <string>
      max_tls_version: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be amazontimestream
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
address	Y	-	The endpoint of the DB instance, from the Timestream for InfluxDB console
port	N	8086, or 8181 for v3	Port of the instance
api_version	N	v2	The write API: v2 for InfluxDB 2, or v3 for InfluxDB 3
token	Y	-	The API token the target signs in with. It needs permission to write to the bucket or the database
timeout	N	30	Seconds one request may take. At least 1

Where the Points Land

Field	Required	Default	Description
org	N*	-	v2 only. The organization the bucket belongs to, by name or ID. Required for v2
bucket	N*	-	v2 only. The bucket. Required for v2
database	N*	-	v3 only. The database. Required for v3

Point Mapping

These four fields apply to every record that the InfluxDB device did not write. See How a Record Becomes a Point.

Field	Required	Default	Description
measurement_field	N	event.dataset	The record field whose text names the measurement
measurement	N	record	The measurement when that field is missing or empty
tags	N	-	The record fields that become tags. A list, or one text with the paths separated by commas
fields	N	-	The record fields that become fields. When empty, every number and boolean in the record

Delivery

Field	Required	Default	Description
batch_size	N	5000	The most lines in one request. AWS suggests about 5,000 lines per request
max_bytes	N	10485760 (10 MB)	The most bytes of line protocol in one request, before compression
field_format	N	-	Converts each record to a format such as ECS or OCSF before it is mapped. See applicable Normalization section

note

max_bytes caps the size of a single request body or batch. It behaves differently from max_size on the file-writing targets, and the difference is easy to get wrong:

• An explicit max_bytes: 0 IS honoured and removes the ceiling on targets that have no ceiling of their own. The value is read with Int64, which returns what you wrote — unlike max_size, which is read with MustInt64 and silently substitutes its default for a zero. Where a target has a fixed ceiling of its own, its page says so and a zero is clamped to that ceiling instead.
• A negative value is rejected at configuration time with max_bytes must not be negative.

The ceiling is validated when the target is built and enforced again on every batch, so a value the receiving service will refuse fails early rather than per request.

TLS

The target always uses HTTPS, because AWS serves its endpoints over TLS only. There is no tls.status to set: the fields below only adjust the connection.

TLS material is resolved through the shared client builder, so these keys mean the same thing on every target that uses it. They are nested under a tls: block.

Field	Required	Default	Description
tls.verify	N	true	Verify the server certificate. On by default — set it to false only to accept an untrusted certificate, and only where you control the network path
tls.server_name	N	-	SNI hostname override for the handshake. Use it when the certificate's name does not match the address you connect to
tls.ca_name	N	-	CA bundle used to verify the server certificate. When unset the host trust store is used; when set it replaces the host trust store rather than adding to it
tls.cert_name	N*	-	Client certificate, for mutual TLS
tls.key_name	N*	-	Client private key, for mutual TLS
tls.passphrase	N	-	Passphrase for an encrypted client private key
tls.min_tls_version	N	tls1.2	Lowest protocol version accepted (tls1.0, tls1.1, tls1.2, tls1.3)
tls.max_tls_version	N	tls1.3	Highest protocol version accepted (tls1.0, tls1.1, tls1.2, tls1.3)

* cert_name and key_name are individually optional but must be supplied together — a certificate without its key, or a key without its certificate, is a configuration error.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The Amazon Timestream for InfluxDB target writes each record as one point of InfluxDB line protocol, and sends each batch in one HTTPS request. The body is gzip compressed. The target sends no precision setting, because both write APIs read nanoseconds by default.

It is the InfluxDB target with the sign-in and the transport that Timestream for InfluxDB requires: a token for both API versions, and HTTPS always. It maps records the same way.

API Versions

api_version	Product	Default port	Where the points land	Sign-in
v2 (default)	InfluxDB 2	8086	org and bucket (both required)	A token (required)
v3	InfluxDB 3	8181	database (required)	A token (required)

How a Record Becomes a Point

• A record that the InfluxDB device wrote keeps its measurement, its tags and its fields. Each field keeps its type: a float, an integer, an unsigned integer, a boolean or a string. The point mapping fields do not apply to it.
• Any other record is mapped. The measurement is the text at measurement_field, or measurement when that field is missing or empty. Each field that tags lists becomes a tag, and each field that fields lists becomes a field, named by its path. When fields is empty, every number in the record becomes a float field and every boolean becomes a boolean field. Numbers in arrays, the record's time and the fields used as tags are left out.
• The time of the point is the record's @timestamp, in nanoseconds. A record without one gets the server's time.
• A record that gives no field is skipped and counted as dropped. The Director logs a warning when no record of a batch gives a field.

Prerequisites

1. A Timestream for InfluxDB instance whose endpoint is reachable from the Director host
2. For v2, the organization and the bucket; for v3, the database
3. An API token with permission to write to the bucket or the database

Examples

InfluxDB 2

Writing to a bucket of a Timestream for InfluxDB instance through the v2 API, the default:

targets:
  - name: timestream_metrics
    type: amazontimestream
    properties:
      address: "a1b2c3d4e5-abcdefghij.timestream-influxdb.us-east-1.on.aws"
      org: "my-org"
      bucket: "metrics"
      token: "your-api-token"

api_version and port are left out, so they default to v2 and 8086.

InfluxDB 3

Writing to a database of an InfluxDB 3 instance through the v3 API:

targets:
  - name: timestream_v3
    type: amazontimestream
    properties:
      address: "a1b2c3d4e5-abcdefghij.timestream-influxdb.us-east-1.on.aws"
      api_version: "v3"
      database: "observability"
      token: "your-api-token"

port is left out, so it defaults to 8181, the port of the v3 API.

Troubleshooting

Situation	Result
The server refuses the data (400, 413 or 422)	The batch is tried up to four times, then dropped. A 400 or 422 answer can mean a partial write: the server keeps the points it accepted, and a point sent again with the same measurement, tags and time merges its fields into the stored point
Sign-in fails (401 or 403), the database or bucket does not exist yet (404), the server is busy (429), a server error (5xx), or the connection fails	The batch is retried
The server answers with a redirect	Counted as a failure: the target does not follow redirects

Records are dropped although nothing fails

Cause: the records give no field. Either they hold no number or boolean, or none of the paths that fields lists has a value in them. When no record of a batch gives a field, the Director logs a warning that starts with Skipped.

Fix: list the record fields that carry the values in fields, or leave fields empty to write every number and boolean.

Configuration errors

These stop the target from starting:

Reason text	Fix
address is required for amazontimestream target	Set address to the endpoint of the DB instance
api_version must be one of v2, v3 for amazontimestream target, got "..."	Set api_version to v2 or v3, or leave it out to use v2
org is required for amazontimestream target with api_version v2	Set org
bucket is required for amazontimestream target with api_version v2	Set bucket
database is required for amazontimestream target with api_version v3	Set database
token is required for amazontimestream target with api_version ...	Set token. Both API versions need one
timeout must be at least 1 second for amazontimestream target, got ...	Set timeout to 1 or more
batch_size must be a whole number greater than 0 for amazontimestream target, got ...	Use a whole number of at least 1, or leave batch_size out to use 5000
failed to build TLS config: ...	The rest of the line names the TLS setting at fault