Skip to main content

Datadog Agent

Synopsis​

Creates a Datadog Agent device. It makes Director the address that Datadog Agents send their logs, metrics and traces to, in place of Datadog. You point your Agents at the device instead of at Datadog. Director can then filter, reduce and route the data before it goes on, to Datadog or to any other target.

The device takes logs, metrics and traces. It does not take service checks or events — see What the Device Does Not Take.

Schema​

- id: <numeric>
  name: <string>
  description: <string>
  type: datadog_agent
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <numeric>
    api_key: <string>
    api_keys:
      - <string>
    accept_logs: <boolean>
    accept_metrics: <boolean>
    accept_traces: <boolean>
    read_timeout: <numeric>
    idle_timeout: <numeric>
    max_connections: <numeric>
    access_list:
      - action: <string>
        ip_blocks: <string>
    tenants:
      - tenant_id: <string>
        ip_blocks: <string>
    max_buffer_size: <string>
    tls:
      status: <boolean>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      min_tls_version: <string>
      max_tls_version: <string>
      client_ca_name: <string>
      client_auth_required: <boolean>

Configuration​

The following fields are used to define the device. No field under properties is required, because the port has a default.

Device​

FieldRequiredDefaultDescription
idY-Unique numeric identifier
nameY-Device name
descriptionN-Optional description
typeY-Must be datadog_agent
tagsN-Optional tags
pipelinesN-Optional pre-processor pipelines
statusNtrueEnable/disable the device

Listener​

FieldRequiredDefaultDescription
addressN0.0.0.0The address to listen on. The default is every interface. The console takes an IPv4 address. In the console: Listen address (Optional)
portN8282The port to listen on. It is a whole number from 1 to 65535. In the console: Port
read_timeoutN30How long a request may take to be read, headers and body. It is a number of seconds, or a duration of whole seconds such as 5m, from 1 to 3600. YAML only
idle_timeoutN120How long a kept-alive connection may wait for its next request. It is from 1 to 86400 seconds, written like read_timeout. YAML only
max_connectionsN10000How many connections may be open at once, from 1 to 100000. The device closes the connection past the limit as it accepts it. YAML only

API Keys​

FieldRequiredDefaultDescription
api_keyN-One API key that the device accepts. It is plain text, ${VARIABLE}, $env{VARIABLE} or $secret{...}. With no api_key and no api_keys, the device takes any request. An empty api_key is not an error: it means the device has no single key. In the console: API key (Optional)
api_keysN-A list of API keys that the device accepts. Each entry is written like api_key. An empty entry is a configuration error. The API does not return the list in plain text. YAML only

See Checking API Keys for how the device reads a request's key.

Data Types​

FieldRequiredDefaultDescription
accept_logsNtrueWhether the device takes logs. A request for a kind that is off gets 202, and the device drops its data and counts it. In the console: Logs
accept_metricsNtrueWhether the device takes metric series and distribution sketches. In the console: Metrics
accept_tracesNtrueWhether the device takes traces and APM stats. In the console: Traces

See Turning a Kind of Data Off.

Access List and Tenants​

FieldRequiredDefaultDescription
access_listN-The source-IP access rules: a list of rules, each with an action (accept or drop) and ip_blocks (addresses or CIDR ranges). The device closes a connection from a source that the list denies. YAML only
tenantsN-The tenant map: a list of entries, each with a tenant_id and ip_blocks. A request from a source IP mapped to a tenant carries that tenant. YAML only

TLS​

FieldRequiredDefaultDescription
tls.statusNfalseTurns TLS on, so that the device serves HTTPS. In the console: TLS encryption
tls.cert_nameY*cert.pemThe certificate that the device serves. The console marks it as required. In the console: Certificate
tls.key_nameY*key.pemThe private key of the certificate. In the console: Private key
tls.passphraseN-The passphrase of the private key, when the key has one. In the console: Passphrase (Optional)
tls.min_tls_versionNtls1.2The lowest TLS version that a connection may use. The console offers TLS 1.0, TLS 1.1, TLS 1.2 and TLS 1.3. In the console: Minimum TLS version (Optional)
tls.max_tls_versionN-The highest TLS version that a connection may use. Leave it out to allow up to TLS 1.3. In the console: Maximum TLS version (Optional)
tls.client_ca_nameN-The CA that signed the certificates the Agents present. The device checks any certificate that an Agent presents against it, whether or not one is required. In the console: CA certificate
tls.client_auth_requiredN**falseMakes the device refuse an Agent that presents no certificate. In the console: Client certificate verification

* = Required when tls.status is true.

** = Requires tls.client_ca_name.

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

  • File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
  • Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
  • Inline PEM content — used verbatim when the value contains -----BEGIN.
  • Environment variable — ${ENV_VAR}.
  • Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

Ingest Buffer​

FieldRequiredDefaultDescription
max_buffer_sizeN128MiBThe ingest admission cap, as on the other device pages. The minimum is 16MiB. A value that is not positive is ignored: it does not mean unlimited. YAML only

This device has five streams: logs, series, sketches, traces and APM stats. The value applies to each stream. When Director runs without crash_resistance (memory mode), the device holds up to 3 times the value for each stream: 384 MiB at the default, so up to 5 × 384 MiB = 1920 MiB in all. See Listener Memory Sizing.

Pointing the Agents at the Device​

Set these in the datadog.yaml of each Agent. All three settings can use the same address:

observability_pipelines_worker:
  logs:
    enabled: true
    url: "https://<director>:8282"
  metrics:
    enabled: true
    url: "https://<director>:8282"
  traces:
    enabled: true
    url: "https://<director>:8282"
  • observability_pipelines_worker.logs.enabled and observability_pipelines_worker.logs.url send the logs.
  • observability_pipelines_worker.metrics.enabled and observability_pipelines_worker.metrics.url send the metric series and the distribution sketches.
  • observability_pipelines_worker.traces.enabled and observability_pipelines_worker.traces.url send the traces and the APM stats.
  • Turn on TLS for the device, and use https://, as above. The Agent sends its own API key in the DD-API-KEY header with every request. Over http://, anyone who can read the traffic sees the key and the data.
  • Use http:// only when the device has no TLS, on a network where nobody else can read the traffic.
  • Port 8282 is the device's default port.

Checking API Keys​

  • With no api_key and no api_keys, the device takes any request. With one of them, the device takes a request only when it carries one of the keys.
  • The Agent sends its key in the DD-API-KEY header. The name of the header can be in any case.
  • When the request has no DD-API-KEY header, the Agent sends the key in the path /v1/input/<api_key>. When a request has both, the header counts.
  • A request with a wrong key, or with none, gets the answer 403 with {"errors":["Forbidden"]}.
  • The device accepts api_key and every entry of api_keys. The console edits api_key, one key. The list is for the configuration file and the API.
  • The device never writes the API key of a request to a record or to a log line. It does store each log as the Agent sent it, so a secret that is part of a log's own content stays in its record.
  • Turn TLS on when you use keys. Without TLS, the device warns when it starts, because the keys travel in the clear.

Paths and Records​

The device takes POST requests only. It answers any other method with 405, and any other path with 404.

PathIt carriesIt becomes
/api/v2/logs, /v1/input, /v1/input/<api_key>Logs: a JSON array of objectsOne record for each object, exactly as the Agent sent it. The device adds nothing
/api/v1/series (JSON), /api/v2/series (protobuf)Metric seriesOne record for each series: dataset datadog_agent.series
/api/beta/sketches (protobuf)Distribution sketchesOne record for each sketch: dataset datadog_agent.sketch
/api/v0.2/traces (protobuf)TracesOne record for each trace chunk: dataset datadog_agent.trace
/api/v0.2/stats (MessagePack)APM statsOne record for each stats payload: dataset datadog_agent.apm_stats
  • A log record is the element as the Agent sent it. The device leaves out only the white space around the element. It leaves out an element that is not an object, and counts it.
  • Every other record is a JSON object with the fields in the next table.
  • Each kind goes to a stream of its own, so you can route each kind on its own.
FieldValue
@timestampThe first point of a series, the first sketch of a sketch record, the earliest span start of a trace chunk, or the first bucket start of a stats payload. When the data has none, the time the request arrived
ecs.version8.11.0
event.moduledatadog_agent
event.datasetOne of the four datasets above
event.kindmetric for series, sketches and stats. event for traces
observervendor is VirtualMetric, product is Datadog Agent, name is the device's name, and hostname is the Director's
source.ipThe address the request came from
datadog_agentThe payload, under series, sketch, trace or apm_stats
  • Under datadog_agent, a series is one Datadog series: the metric, its type, points, tags and resources.
  • A sketch is one Datadog sketch. A trace record is one trace chunk, with the information of its tracer and of the Agent. A stats record is one stats payload.
  • The ids and the start times of a span, and the start time of a stats bucket, are strings. A reader that parses every number as a float would round them.

Turning a Kind of Data Off​

  • Each kind is on until you turn it off with accept_logs, accept_metrics or accept_traces.
  • accept_metrics governs the series and the sketches. accept_traces governs the traces and the APM stats.
  • The device answers a request for a kind that is off with 202 and {}. It drops the data and counts the request as dropped. The Agent sees a success, so it does not send the data again.
  • The device reads the body of such a request and throws it away without decoding it. It reads the body through the 10 MiB limit, as for any request, and answers 413 past it.
  • A request still needs a correct API key.
  • When Director routes nothing, the device answers 202 and drops the data in the same way. It reads the body, throws it away and counts the request as dropped.

Answers​

AnswerWhen
202 with {}The device accepted the request. The device answers only after the records it kept are stored. A 202 does not mean that no record was dropped. The device counts what it leaves out, such as values that JSON cannot hold. It drops a whole request whose kind is off, or that comes while Director routes nothing
400The body does not decode: it is not a JSON array (logs), or not a valid payload of its kind, or its compression is damaged
403The key is wrong or missing. It is also the answer to a source that the access list denied after you edited the list while the device was running, and the device then closes the connection. A source that the access list denies when it connects has its connection closed as the device accepts it, so it gets no answer. The body is {"errors":["Forbidden"]}
404The path is not one of the paths above
405The method is not POST. The Allow header says POST
413The body is too large, holds too many items, or needs more memory to decode than the device has (see Limits)
415The Content-Encoding is not one the device takes
500The device failed on the request, or could not write its records
503Director cannot take the records now: its ingest buffer is full, it has no memory to spare for the request, or it is stopping. The Agent sends the request again

Every error body is {"errors":["..."]}, as Datadog's is. The exact messages are under What the Agent Sees.

Limits​

  • Compression. The device takes gzip, deflate (zlib), zstd and no encoding. It answers any other encoding with 415.
  • Size. A body may have at most 10 MiB as it is sent, and at most 64 MiB once it is decompressed. The device answers a larger body with 413. It never decompresses without a limit. The limit for a body as it is sent also applies to a body that the device drops, because its kind is off or Director routes nothing.
  • Items. One request may hold at most 250,000 items in all. The limit is one total for the whole payload. The device counts every item that it decodes, and it counts the items inside other items, tags included.
    • For series, the items are the series, their points, their resources and their tags.
    • For sketches, they are the sketches, their entries and their tags.
    • For traces, they are the tracer payloads, their chunks, their spans, the links and events of the spans, and the values of the attribute lists.
    • For APM stats, they are the clients, their buckets, the grouped stats and the tags of the clients and of the grouped stats.
    • The device answers a request with more with 413, and counts it as dropped. Logs have a separate limit of 100,000 entries in one request. Real Agents stay far below these limits.
  • Memory. All the bodies in flight together may use 256 MiB. Before it decodes a metrics, traces or stats request, the device counts its items and reserves memory for the body and for each item. An item of series takes 320 bytes, of sketches 448, of traces 768 and of APM stats 640. The device reserves at least eight times the size of the body.
    • A request that needs more than the whole 256 MiB, with the memory its body already uses, gets 413 at once.
    • A request that has to wait for memory waits at most 30 seconds, and then gets 503.
  • Time. read_timeout, idle_timeout and max_connections limit the requests and the connections (see Listener).

The device does not allocate memory from a length that a request claims. Its buffers grow as the bytes arrive.

What the Device Leaves Out​

The device counts what it leaves out. It warns about each kind of drop at most once a minute, with how many. It never logs what was in the data. When the device stops, it logs the totals.

  • A log element that is not an object.
  • A record that is larger than the ingest buffer takes (max_buffer_size).
  • A series point, a sketch entry or a span metric whose number is not finite. JSON cannot hold NaN or an infinity. The same goes for an Agent's sampling rate and an attribute of a span event. A series or a sketch that is left with nothing is not written.
  • A tracer payload in the newer compact form. The device does not read it yet.
  • A legacy distribution in a sketch payload. The device does not read it.
  • A record that cannot be made into JSON.
  • A whole request: for a kind that is turned off, that comes while Director routes nothing, with a body that does not decode, or with too many items.

What the Device Does Not Take​

The device does not take service checks or events. The Agent sends them only to its main Datadog address, not to the observability_pipelines_worker address. They keep going to Datadog directly.

Sending the Data On​

Back to Datadog. The Datadog target sends the records of this device back to Datadog in Datadog's own formats.

To other metric targets. The OTLP and Prometheus Remote Write targets send series and sketch records as metrics. They get these points:

  • A series gives one point for each point of the series. The name is the metric name, and the unit is the unit of the series, as the Agent sent it.
  • A count is a delta counter. A gauge, a rate and a series with no type are gauges. The time is the point's own time.
  • A sketch gives, for each entry of the sketch, <metric>.count as a delta counter, and <metric>.sum, <metric>.min, <metric>.max and <metric>.avg as gauges. The sum is a gauge because it can be negative.
  • A time later than the year 2262 cannot be sent in nanoseconds. A point with such a time is left out. So are all the points of a sketch entry with such a time.
  • The attributes of a point are the record's labels, the tags of the series or the sketch, and host. A tag key:value gives the key and the value. A tag with no colon gives the tag as the key and an empty value. The host is the series' host resource, or the sketch's host.
  • Traces and APM stats give no points. The targets send these records as they send any other record.

Changing the Configuration​

  • The device starts again with the new configuration when you change the address, the port, the TLS settings, a switch, a key, a timeout or the connection limit.
  • An edit that no longer validates starts the device again too. The start then reports why it cannot run.
  • A key that names a secret Director cannot read just now does not restart the device. The device keeps running with the keys it read.
  • An edit of the access list or the tenants reaches a running device at once. A device that started without one starts again when you add the first one.

Clusters​

Every node of a cluster runs the device. An Agent sends to one node, or to the load balancer in front of the nodes.

Console​

In the console, go to Fleet Management > Devices > Datadog Agent, listed under Protocols, and click Add new device. The wizard's steps are:

  1. General settings, as on every device.
  2. Protocol settings. The step's description says: Datadog Agents send their data to the address and port below. Point each Agent at this director in place of Datadog, and it sends its logs, metrics and traces as it would to Datadog. Every node of a cluster listens.
  3. Review. It shows the address, the port, the API key, the three data types and the TLS settings. It shows the API key, the certificate, the private key and the CA certificate as set or not set, never as themselves.

The Protocol settings step has these fields:

  • Listen address (Optional) and Port share one row. The placeholder of the address is 0.0.0.0, and its helper text says: The address the listener binds. Defaults to 0.0.0.0, every interface. The placeholder of the port is Enter the port, and its helper text says: The listener binds this port, and Datadog Agents send their data to it. The port is a number from 1 to 65535. Defaults to 8282.
  • API key (Optional), a secret field. It takes plain text, an environment variable or a vault secret. Its placeholder is Enter the API key. Its helper text says: Datadog Agents must send this API key with their data. While this field is empty, the device accepts any key. On a saved device, an empty field keeps the key, and Remove lets any key in.
  • A warning notice, The API key is sent in cleartext. It shows while an API key is set and TLS encryption is off. Its subtitle says: Over plain HTTP, Datadog Agents send their API key unencrypted with every request, and anyone who reads it can send data to the device. Turn TLS encryption on, or use an API key only where the network between the Agents and the director is trusted.
  • Under Data types, described as Choose the kinds of data that the device takes from Datadog Agents. The device accepts the data of a kind that is turned off, drops it and counts it, so the Agent does not send it again.: the switches Logs, Metrics and Traces. Each is Disabled or Enabled, and each is enabled by default.
  • Under TLS settings: the TLS encryption switch, which is off by default. Its helper text says: When enabled, the device serves HTTPS with the certificate below, and can require Agents to present certificates of their own. When disabled, it serves plain HTTP.

With TLS encryption on, the fields come in this order:

  1. An info notice, Certificate chain and trust requirements. It says: If an intermediate CA issued the certificate, include the full chain: the certificate first, then each intermediate. The other end of the connection must trust the issuing CA, so install the CA certificate on machines that do not trust it yet.
  2. Certificate, with a Value source picker and an Upload certificate link.
  3. Private key, with a Value source picker and an Upload private key link.
  4. Passphrase (Optional), with the placeholder Enter the passphrase.
  5. The Client certificate verification checkbox. Its helper text says: Enables mutual TLS: every client must present a certificate that verifies against the CA below.
  6. While it is checked, CA certificate, with a Value source picker and an Upload CA certificate link.
  7. Last, Minimum TLS version (Optional) and Maximum TLS version (Optional) in one row. Both show Select a TLS version as the placeholder. The helper text of the minimum says: The lowest TLS version the connection may use. Defaults to TLS 1.2. The helper text of the maximum says: The highest TLS version the connection may use. Leave empty to allow up to TLS 1.3. On the device page, an empty minimum reads Default (TLS 1.2).

The certificate, the private key and the CA certificate each take the text, an environment variable, a Vault secret or a file path.

A switch can hold an environment variable in the YAML. The form then shows the variable in a text field in place of the switch. The placeholder says e.g. ${DATADOG_AGENT_SETTING}, and the helper text says: The director reads this switch from this environment variable where it runs, true or false, which the console cannot show. Clear the field to use the switch instead. For Client certificate verification, the helper text of that field says: The director reads this setting from this environment variable where it runs, true or false, in place of the switch above, which the console cannot show. Clear the field to use the switch instead.

The form checks the step as Director checks the configuration when it starts, so a device that saves is a device that starts. In the messages below, <value> stands for what the field holds and <size> for a size:

  • The address: Listen address must be an IPv4 address, or 0.0.0.0 for every interface.
  • The port: Port is required, and Port must be a whole number from 1 to 65535.
  • A switch that holds something other than blank or an environment variable: The director reads this switch as on or off, not as <value>: clear the field to choose with the switch, or name an environment variable such as ${DATADOG_AGENT_SETTING}.
  • With TLS encryption on, the certificate: Certificate is required when it is empty, Provide a valid certificate to continue when the text is not a certificate, This certificate has expired. Provide a valid certificate to continue, and This certificate is larger than <size>. Provide a smaller certificate to continue.
  • With TLS encryption on, the key: Private key is required.
  • The TLS versions: TLS version must be TLS 1.0, TLS 1.1, TLS 1.2 or TLS 1.3, and Maximum TLS version must not be lower than Minimum TLS version (TLS 1.2 when empty).
  • Client certificates: Provide a CA certificate to verify client certificates against. It shows when client certificates are required and no CA is given.

The form has no fields for api_keys, access_list, tenants, read_timeout, idle_timeout, max_connections and max_buffer_size. They are YAML only. A device that has api_keys, access_list, tenants, read_timeout, idle_timeout or max_connections in its YAML keeps them when you edit the device in the console.

Troubleshooting​

Configuration Errors​

The Director refuses a configuration it cannot use, with one of these messages. <name> stands for the name of the device, <key> for the name of a field, <reason> for the reason the Director gives and N for a number. The log line is Can not validate listener config for <name>. Error: <message>, and the connection state of the device says Configuration validation failed for <name>: <message>:

  • address not valid: the address cannot be used. The Director checks the port first, so a bad port is reported as one.
  • <key> is not a whole number: <key> is port, read_timeout, idle_timeout or max_connections.
  • port must be from 1 to 65535, not N
  • read_timeout must be from 1 to 3600, not N
  • idle_timeout must be from 1 to 86400, not N
  • max_connections must be from 1 to 100000, not N
  • <key> is <value>: it is true or false: <key> is accept_logs, accept_metrics, accept_traces, tls.status or tls.client_auth_required. The switch holds a value that is neither true nor false.
  • tls: the certificate or the private key does not load
  • tls: the private key does not decrypt with its passphrase: <reason>
  • tls: the certificate and the private key are not a pair: <reason>
  • tls: min_tls_version is above max_tls_version
  • tls.client_auth_required needs tls.client_ca_name: the CA that signed the certificates the Agents present
  • tls.client_ca_name names no CA certificate that loads
  • tls.client_ca_name holds no certificate in PEM form
  • api_key is one secret, written as text
  • api_keys is a list of secrets
  • api_keys: entry N is not text: it names a secret
  • api_keys: entry N is empty: an Agent that sends no key would be let in
  • api_key resolves to nothing: an Agent that sends no key would be let in. For an entry of the list, the message starts with api_keys entry N.
  • a secret could not be resolved: <what> of device "<name>": <reason>: a variable that is not set, or a vault that does not answer. <what> is api_key, api_keys entry N, or the TLS certificate, the TLS private key, the TLS passphrase or the TLS clients' CA. For a TLS secret, the message reads a secret could not be resolved: the TLS certificate of device "<name>": <reason>.

Errors When the Device Starts​

These appear in the Director's log. The device does not start.

  • Can not start DatadogAgent listener for <name>. Error: <reason>: the port is taken, or the TLS files do not load. The connection state of the device says Connection validation failed for <name>: <reason>.
  • Can not start the listener of <name>. Error: <reason>: the device could not bind its address. The connection state of the device says Can not start the listener of <name>: <reason>.
  • Failed to run the DatadogAgent listener of <name>. Error: <reason>: the server stopped with an error.
  • Skipping initialization of listener.: the device is turned off, so the Director does not start it. It is an information line.

Log Lines While It Runs​

These appear in the Director's log. <name> stands for the name of the device, <client> for the address the request came from and N for a count. The Director logs each line below at most once a minute for each device and each kind of line. When it holds lines back, the next line ends with (N more like it since this was last logged). The lines of the start, the stop and the errors of the start are not held back.

Information:

  • Listener (<name>) server is successfully started on <address> over <http or https>, serving the Datadog Agent API (logs <on or off>, metrics <on or off>, traces <on or off>) <with API keys or without API keys>.
  • Listener (<name>) stopped. Records: N received, N written. Left out: N elements that are not objects, N records too large for the ingest buffer, N records that could not be made into JSON, N values that are not finite numbers, N compact tracer payloads, N legacy distributions. Requests: N dropped because their data type is turned off, N dropped because the director routes nothing, N dropped because their body could not be decoded, N dropped because they held too many items.
  • Dropped a request from <client> to <name>: the device does not take <logs, metrics or traces> (accept_<logs, metrics or traces> is false).
  • Dropped a request from <client> to <name>: the director routes nothing.

Warnings:

  • Agents send their API key over HTTP without TLS: it travels in the clear (turn TLS on) (<name>): the device has keys and no TLS. The Director logs it once, when the device starts.
  • Refused a request from <client> to <name>: it sent no API key, or one the device does not take.
  • Closed a connection from <client> to <name> as it was accepted: the access list denies it.
  • Closed a connection from <client> to <name>: N connections are open, as many as max_connections takes.
  • Refused a request from <client> to <name>: it holds more items than the device reads in one request (250000 decoded items in all, or 100000 logs).
  • Refused a request from <client> to <name>: decoding it needs more memory than the device has for all requests (268435456 bytes).
  • Refused a request from <client> to <name>: its body is too large (10485760 bytes as sent, 67108864 bytes once decompressed, at most 100000 records).
  • Refused a request from <client> to <name>: its Content-Encoding is not one the device decodes.
  • Refused a request from <client> to <name>: its body is not a JSON array, or could not be read.
  • A request from <client> to <name> was answered 503: the device has no memory to spare for it, or is stopping. The Agent sends it again.
  • A request from <client> to <name> waited 30s for the device's ingest buffer, and was answered 503 with N of its records written, which a retry writes again.
  • Left N elements out of a request from <client> to <name>: they are not JSON objects.
  • Dropped N records of a request from <client> to <name>: each is larger than the device's ingest buffer admits (max_buffer_size).
  • Dropped N records of a request from <client> to <name>: they could not be made into JSON.
  • Left N values out of a request from <client> to <name>: JSON cannot hold a number that is not finite.
  • Left N tracer payloads out of a request from <client> to <name>: the device does not read tracer payloads in their compact form yet.
  • Left N legacy distributions out of a request from <client> to <name>: the device does not read them.
  • The listener of <name> did not stop within 10s: the writes it still waits on are dropped.
  • <name> did not stop within 30s; its collector waits for it.

Errors:

  • Can not write the records of a request from <client> to <name>, N of them written. Error: <reason>: the device answered 500.
  • Can not write the records of a request from <client> to <name> to disk, N of them written. The request was answered 503, which a retry writes again. Error: <reason>
  • A request from <client> to <name> failed: <reason>. Stack: <stack>: the device answered 500.
  • DatadogAgent heartbeat check failed. Stopping collector...: the device stopped answering its health check, and the Director stops it.

Debug:

  • The HTTP server of <name>: <message of the HTTP server>

What the Agent Sees​

Every error body is {"errors":["<message>"]}. The messages are:

AnswerMessage
400the body is not a JSON array (logs); the body is not a Datadog series payload, the body is not a Datadog sketch payload, the body is not a Datadog trace payload or the body is not a Datadog APM stats payload (the other kinds); the body could not be decompressed: <reason>; and the body could not be read
403Forbidden
404Not Found
405Method Not Allowed
413the body is larger than the device takes: 10485760 bytes as sent, 67108864 bytes once decompressed. Or: the request needs more memory than the device has for all requests: 268435456 bytes. Or: the payload holds more than 250000 items. Or, for logs: the body holds more than 100000 records
415the Content-Encoding is not one the device takes: gzip, deflate (zlib), zstd or none
500internal error, or the records could not be written
503the director is busy: send the data again later, or the director could not store the records: send the data again later, or the device is stopping: send the data again later

Examples​

Basic​

Datadog Agents send their logs, metrics and traces to port 8282, and must send the API key. Use this only on a network where nobody else can read the traffic...

- id: 1
  name: datadog-agent
  description: Datadog Agents sending their logs, metrics and traces
  type: datadog_agent
  properties:
    port: 8282
    api_key: "${DATADOG_AGENT_API_KEY}"

TLS and a Key for Each Fleet​

The same over TLS, with a key for each Agent fleet and no traces. An entry of the list can be a vault secret as well. To check the certificates of the Agents (mTLS), name the CA that signed them; client_auth_required refuses an Agent that presents none...

- id: 2
  name: datadog-agent-tls
  description: Datadog Agents sending logs and metrics over TLS, with a key for each fleet
  type: datadog_agent
  properties:
    tls:
      status: true
      cert_name: "datadog-agent.example.com.crt"
      key_name: "datadog-agent.example.com.key"
      min_tls_version: "tls1.2"
      # client_ca_name: "agents-ca.crt"
      # client_auth_required: true
    api_keys:
      - "${DATADOG_AGENT_API_KEY_PROD}"
      - "$env{DATADOG_AGENT_API_KEY_STAGING}"
    accept_traces: false

Logs Only​

Only logs. The access list lets in only the 10.0.0.0/8 network, and a request from 10.1.0.0/16 carries the tenant acme...

- id: 3
  name: datadog-agent-logs
  description: Datadog Agents sending only their logs
  type: datadog_agent
  properties:
    api_key: "${DATADOG_AGENT_API_KEY}"
    accept_metrics: false
    accept_traces: false
    access_list:
      - action: accept
        ip_blocks: "10.0.0.0/8"
    tenants:
      - tenant_id: "acme"
        ip_blocks: "10.1.0.0/16"