Fluent Forward

Synopsis

Creates a Fluent Forward device. It makes Director the Fluentd aggregator that Fluent Bit and Fluentd send their events to. The forward output of Fluent Bit, the out_forward of Fluentd and the fluent-logger libraries send to it as they would send to Fluentd.

The device speaks the Fluent forward protocol (v1) over TCP, with TLS when you turn it on. It writes every event of a message as one log record. It aggregates nothing. It sends back only the handshake and the acknowledgements that a sender asks for.

The device takes logs only, and it drops the metrics and traces that Fluent Bit sends — see Metrics and Traces.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: fluent_forward
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <numeric>
    shared_key: <string>
    users:
      - username: <string>
        password: <string>
    username: <string>
    password: <string>
    self_hostname: <string>
    timeout: <numeric>
    max_connections: <numeric>
    max_message_bytes: <string>
    access_list:
      - action: <string>
        ip_blocks: <string>
    tenants:
      - tenant_id: <string>
        ip_blocks: <string>
    max_buffer_size: <string>
    tls:
      status: <boolean>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      min_tls_version: <string>
      max_tls_version: <string>
      client_ca_name: <string>
      client_auth_required: <boolean>

Configuration

The following fields are used to define the device. No field under properties is required, because the port and the limits have defaults. The device does not read the listener settings workers, reuse, framing, encoding and proxy_protocol, because it runs one server on a socket of its own.

Device

Field	Required	Default	Description
id	Y	-	Unique numeric identifier
name	Y	-	Device name
description	N	-	Optional description
type	Y	-	Must be fluent_forward
tags	N	-	Optional tags
pipelines	N	-	Optional pre-processor pipelines
status	N	true	Enable/disable the device

Listener

Field	Required	Default	Description
address	N	0.0.0.0	The address to listen on. The default is every interface. The console takes an IPv4 address. In the console: Listen address (Optional)
port	N	24224	The port to listen on. It is a whole number from 1 to 65535. The default is the port of a Fluentd aggregator, so senders keep their usual port. In the console: Port
timeout	N	600	How many seconds a connection may wait for its next message. A client with nothing to send keeps its connection open. 0 means no limit. YAML only
max_connections	N	10000	The most connections that can be open at once. The device closes each new connection past the limit as it accepts it. 0 is read as the default. YAML only
max_message_bytes	N	32MB	The largest message: a whole number of bytes, or a size such as 64MB, from 1 KiB to 1 GiB. It bounds a message as it is sent, its payload once decompressed, and the records built from it. The device drops a message past it and closes the connection. The console takes a whole number of bytes. In the console: Maximum message size (Optional)

Authentication

Field	Required	Default	Description
shared_key	N	-	The key that every client must know. Setting it turns the handshake on. Without it, the device sends nothing and reads messages at once. It is plain text, ${VARIABLE} or $secret{...}. In the console: Shared key
users	N	-	A list of users, each with a username and a password. A client must also sign in as one of them. The users need a shared_key. Each password is plain text, ${VARIABLE} or $secret{...}. YAML only
username	N	-	One more user, beside the users list. The console sets this one. It needs a shared_key and a password, and its name must differ from every name in users. In the console: Username (Optional)
password	N	-	The password of that user. It needs a username. It is plain text, ${VARIABLE} or $secret{...}. In the console: Password (Optional)
self_hostname	N	The host name of the Director	The name that the Director gives in its answer to the handshake. YAML only

TLS

Field	Required	Default	Description
tls.status	N	false	Turns TLS on, so that the device serves TLS. It must be true or false. In the console: TLS encryption
tls.cert_name	Y*	-	The certificate that the device serves. The console marks it as required. In the console: Certificate
tls.key_name	Y*	-	The private key of the certificate. In the console: Private key
tls.passphrase	N	-	The passphrase of the private key, when the key has one. In the console: Passphrase (Optional)
tls.min_tls_version	N	tls1.2	The lowest TLS version that a connection may use. The console offers TLS 1.0, TLS 1.1, TLS 1.2 and TLS 1.3. In the console: Minimum TLS version (Optional)
tls.max_tls_version	N	-	The highest TLS version that a connection may use. Leave it out to allow up to TLS 1.3. In the console: Maximum TLS version (Optional)
tls.client_ca_name	N	-	The CA that signed the certificates that clients present (mTLS). It must be a CA certificate in PEM form. In the console: CA certificate
tls.client_auth_required	N	false	Makes the device refuse a client that presents no certificate. It must be true or false. The console asks for the CA certificate when it is on. In the console: Client certificate verification

* = Required when tls.status is true.

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

• File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
• Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
• Inline PEM content — used verbatim when the value contains -----BEGIN.
• Environment variable — ${ENV_VAR}.
• Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

Access List

The access_list property is a source-IP firewall: an ordered list of accept/drop rules over IP blocks, evaluated before the listener parses or authenticates the connection. When it is not set, the firewall is off and adds no overhead.

Field	Required	Default	Description
access_list	N	-	Ordered list of source-IP rules. Omit to disable the firewall.
access_list[].action	Y		accept or drop (synonyms: allow/permit and deny/reject/block). Unrecognized actions are ignored.
access_list[].ip_blocks	Y		One or more IP blocks the rule matches. Comma-separated string or a list. Accepts CIDR (10.0.0.0/24), single IP (172.16.5.10), dotted-netmask (192.168.1.0/255.255.255.0, IPv4 only), and inclusive range (10.5.0.1-10.5.0.50). IPv4 and IPv6.
access_list_default	N	auto	Verdict for a source IP that matches no rule: accept or drop. When omitted, it is derived automatically (see below).

Rules are evaluated top to bottom and the first rule whose blocks contain the source IP decides the outcome. A source IP that matches no rule follows the default policy:

• If any accept rule is present, the list is treated as an allowlist and unmatched IPs are dropped.
• Otherwise the list is treated as a denylist and unmatched IPs are accepted.

Set access_list_default to override this. An unparseable source IP is judged by the default policy, so an allowlist fails closed.

access_list:
  - action: accept
    ip_blocks: "10.0.0.0/8, 192.168.0.0/16, 172.16.5.10"

note

Editing or removing access_list rules applies on the next configuration reconcile with no listener restart. Enabling access_list for the first time on a device that had none takes effect on the next collector restart.

The device closes a connection from a denied address as it accepts it. In the console: Source IP filtering.

Tenants

The tenants property maps a source IP to a tenant: an ordered list of rules matched against the connection's (or datagram's) source address before the record enters the pipeline. Multiple IP blocks can map to the same tenant, and multiple tenants are supported. When it is not set, no tenant is attached.

Field	Required	Default	Description
tenants	N	-	Ordered list of source-IP-to-tenant rules. Omit to disable IP-based tenancy.
tenants[].tenant_id	Y		Tenant identifier attached to records arriving from the matching IP blocks.
tenants[].ip_blocks	Y		One or more IP blocks the rule matches. Comma-separated string or a list. Accepts CIDR (10.0.0.0/24), single IP (172.16.5.10), dotted-netmask (192.168.1.0/255.255.255.0, IPv4 only), and inclusive range (10.5.0.1-10.5.0.50). IPv4 and IPv6.

Rules are evaluated top to bottom and the first rule whose blocks contain the source IP wins. A source IP that matches no rule receives no tenant — the record is still ingested. When a tenant is resolved it is written to _vmetric.event.tenant_id; the client IP always stays at _vmetric.event.request. Use _vmetric.event.tenant_id to route or isolate each customer's data downstream (see Routes).

tenants:
  - tenant_id: acme
    ip_blocks: "10.0.0.0/24, 10.1.2.3, 192.168.10.0/255.255.255.0"
  - tenant_id: globex
    ip_blocks: "10.2.0.0/16, 172.16.5.1-172.16.5.50"

note

Editing or removing tenants rules applies on the next configuration reconcile with no listener restart. Enabling tenants for the first time on a device that had none takes effect on the next collector restart.

In the console: Tenant mapping.

Ingest Buffer

Field	Required	Default	Description
max_buffer_size	N	128MiB	The ingest admission cap, as on the other device pages. The minimum is 16MiB. A value that is not positive is ignored: it does not mean unlimited. YAML only

When Director runs without crash_resistance (memory mode), the device holds up to 3 times the value: 384 MiB at the default. With crash_resistance, the value only narrows admission. See Listener Memory Sizing.

Pointing Fluent Bit and Fluentd at the Device

Replace director.example.com with the address of the Director. Port 24224 is the default port of the device.

Fluent Bit, with no handshake and no TLS:

[OUTPUT]
    Name   forward
    Match  *
    Host   director.example.com
    Port   24224

Fluent Bit, with acknowledgements and gzip. The device acknowledges a message when the sender asks for it:

[OUTPUT]
    Name                  forward
    Match                 *
    Host                  director.example.com
    Port                  24224
    Require_ack_response  On
    Compress              gzip

Fluent Bit, with the shared key and one user:

[OUTPUT]
    Name           forward
    Match          *
    Host           director.example.com
    Port           24224
    Shared_Key     the-shared-key
    Self_Hostname  fluent-bit-1
    Username       fluent-bit
    Password       the-password

Fluent Bit, with TLS. Add tls.crt_file and tls.key_file when the device asks for client certificates:

[OUTPUT]
    Name         forward
    Match        *
    Host         director.example.com
    Port         24224
    tls          On
    tls.verify   On
    tls.ca_file  /etc/fluent-bit/ca.crt

Fluentd, with acknowledgements and gzip:

<match **>
  @type forward
  require_ack_response true
  compress gzip
  <server>
    host director.example.com
    port 24224
  </server>
</match>

Fluentd, with the shared key and one user:

<match **>
  @type forward
  <security>
    self_hostname fluentd-1
    shared_key the-shared-key
  </security>
  <server>
    host director.example.com
    port 24224
    username fluentd
    password the-password
  </server>
</match>

Fluentd, with TLS:

<match **>
  @type forward
  transport tls
  tls_cert_path /etc/fluent/ca.crt
  tls_verify_hostname true
  <server>
    host director.example.com
    port 24224
  </server>
</match>

• Fluentd checks that a server is alive with heartbeats. The device does not answer UDP heartbeats.
• Leave heartbeat_type at its default, transport, or set it to none. With udp, Fluentd marks the Director as down.

The Protocol

Message Modes

The device reads all four:

Mode	A message holds
Message	One event
Forward	Many events
PackedForward	Many events in one packed payload
CompressedPackedForward	Many events in a gzip payload. The payload may hold several gzip members, and the device reads them all

• A message may carry a chunk, which is how a sender asks for an acknowledgement. The device reads gzip compression only. Another compression makes the message malformed.
• The device reads the time of an event as whole seconds, as seconds with a fraction, or as Fluentd's EventTime.
• Any other form of time makes the message malformed, such as a list that holds a time and other data. So does a time before year 1 or after year 9999.

Acknowledgements

• A sender asks for an acknowledgement with a chunk. The device then answers {"ack": chunk}.
• The device answers only after Director has stored every event of the message. In crash-resistant mode, the events are then on disk. On a cluster that confirms on majority, the cluster's queue then holds them.
• The device answers all the messages that it read in one go together, after one wait.
• When Director cannot store the events, the device sends no answer and closes the connection, so the sender sends the message again.
• A message with no chunk gets no answer.

The Handshake

It runs only when shared_key is set. Without a key, the device sends nothing and reads messages at once.

1. The device sends HELO as soon as a client connects. HELO holds a random nonce, and a salt when the device has users.
2. The client answers PING within 60 seconds. The PING proves that the client knows the shared key and, when the device has users, the password of one of them.
3. The device answers PONG. To let the client in, it sends a digest that proves to the client that the Director knows the key. To refuse the client, it sends the reason shared_key mismatch or username/password mismatch, counts the failure and closes the connection. A client whose first message is not a PING is refused the same way, with the reason the message is not a PING.

warning

Digests do not hide the data. Turn TLS on when the network is not private.

Records

The device writes one log record for each event. It writes the record as JSON once it has read the whole message:

{
  "@timestamp": "2026-10-29T10:15:30.123456789Z",
  "log": "GET /index 200",
  "stream": "stdout",
  "fluent": { "tag": "app.web" }
}

• Every key of the event's record is a key of the log record, at the top level.
• @timestamp is the time of the event, as RFC 3339 with nanoseconds, in UTC.
• fluent.tag is the tag of the message.
• @timestamp and fluent.tag replace keys of the same name that the sender sent. When the event holds a map under fluent, its other keys stay, and only its tag is set.
• Each record carries the address of the client. It carries the tenant too, when the tenant map names one for that address.
• The records do not name the user that a client signed in as.

How the device turns the values of an event into JSON:

In the event	In the record
A string, or binary data	A string. A byte that is not UTF-8 becomes U+FFFD
An integer	An integer of the same size, signed or unsigned
A float	A number that stays a float (2.0, never 2). A NaN and an infinity become null
A boolean, nil	true or false, null
A map key that is not a string	The text of the key: 5, true, null, [1,2]
An EventTime inside a record	An RFC 3339 string with nanoseconds
Any other extension	null

Limits

• max_message_bytes bounds a message as it is sent, the payload of a PackedForward message once decompressed, and the records built from it. A gzip member that grows past it is refused, so a small message cannot grow into a large one. The device drops a message that is past it.
• Nesting. A record may nest arrays and maps 64 levels deep, the record being the first level. A deeper record makes the message malformed.
• A bad message. The device counts it as dropped, warns at most once a minute and closes the connection. It stores and acknowledges the messages that the client sent before it. It reads a message in full before it writes the first event, so a bad message leaves no events behind.
• A record that is too large for the ingest buffer. The device drops and counts it. It still acknowledges the message, because sending the record again could never work. The same holds for a record that passes the device's own check but that the Director's ingest limit refuses.
• A chunk longer than 1 KiB, a tag longer than 64 KiB and a PING longer than 64 KiB are malformed. The device closes the connection without reading the rest.
• Time. A connection waits timeout seconds for its next message. The device writes each reply to a client within 60 seconds.

Metrics and Traces

Fluent Bit sends the metrics and traces that it collects through the same output, and marks them as such. The device takes logs only. For a message with metrics or traces, the device:

• does not read the entries of the message,
• counts the events of the message as dropped, as many as the message says, or one when it says nothing,
• warns at most once a minute that it takes logs only,
• sends the acknowledgement when the sender asked for one, and
• keeps the connection open.

Without the acknowledgement, Fluent Bit would send the same message again and again. Fluent Bit sends metrics and traces as a PackedForward payload, which the device never reads.

When Director routes nothing, the device still accepts the messages, acknowledges them and drops their events.

When Things Happen

• Start. The device reads its configuration and loads the TLS material while it checks it. Then it binds the port once to check that it is free, and reports Connected. Then it binds the port for good.
• A connection. The device checks the access list first. With a shared_key, it runs the handshake. Then it reads messages.
• A message. The device reads all of it, writes its events, waits until Director has stored them, and sends the acknowledgement.
• Stop. The device closes its socket and the connections that are open. The senders connect again.
• Counters. At each statistics interval, the device logs the events that it received, wrote and dropped, and the clients that it refused at the handshake. It logs the drops as an error too.

Changing the Configuration

• The device starts again when you change the address, the port, the timeouts or the limits. It also starts again when you change the shared key, the users (a rotated secret counts), the host name or the TLS settings.
• The device starts again when you add the first access_list or the first tenants to a device that started without one. Later edits of those two reach the running device at once.
• An edit that no longer validates starts the device again too. The start then reports why it cannot run.
• A key or a password that names a secret the Director cannot read just now does not restart the device. The device keeps running with what it read.

Clusters

Every node of a cluster, and every replica of a container Director, runs the device and binds the port. A sender sends to the node that it is configured with, or to the load balancer in front of the nodes. The console's reachability check reads the port as TCP.

What the Device Does Not Do

• It answers no UDP heartbeat, and it has no Unix socket.
• It takes logs only. It drops metrics and traces.
• It reads gzip only.
• It has no simulated data. Its records are the events that its senders send.

Console

In the console, go to Fleet Management > Devices > Fluent Forward, listed under Protocols, and click Add new device. The wizard's steps are:

1. General settings, as on every device.
2. Protocol settings: the address, the port, the shared key and TLS.
3. Access settings: the access list.
4. Advanced settings: the largest message and the tenants.
5. Review.

The TCP, Syslog and Splunk S2S devices have the same access list and tenants blocks — see TCP, Syslog and Splunk S2S.

Protocol Settings

The step's description says: Fluent Bit, Fluentd and other Fluent Forward clients send their logs, compressed or not, to this director's address and port, as they would to a Fluentd aggregator. The director writes each event as a log record.

• Listen address (Optional) and Port share one row. The placeholder of the address is 0.0.0.0, and its helper text says: The address the listener binds. Defaults to 0.0.0.0, every interface. The placeholder of the port is Enter the port, and its helper text says: The port the listener binds, which clients send to, from 1 to 65535. Fluent Bit and Fluentd send to 24224 unless told otherwise.
• A Shared key section. Its description says: A shared key lets the director and each client confirm that they share the same key before the client sends logs. The key and the password accept plain text, an environment variable or a vault secret. Without a shared key, clients do not need a key. It holds the Shared key authentication switch, which is Disabled or Enabled. With the switch enabled, the section shows two more rows:
  • Shared key, a secret field. Its placeholder is Enter the shared key. Its helper text says: Fluent Bit and Fluentd clients must use this key as their shared key. It is required while shared key authentication is on.
  • Username (Optional) and Password (Optional) share one row. The placeholder of the username is e.g. fluent, and its helper text says: Clients sign in as this user in addition to using the shared key. The username is required when you set a password. The password is a secret field with the placeholder Enter the password. Its helper text says: Clients sign in with this password. It is required when you set a username. Leave both fields empty if clients only need the shared key.
  • Turning the switch off removes the shared key, the username and the password from the device when you save. A users list in the YAML stays, and a device that has users and no shared key does not start.
• Under TLS settings: the TLS encryption switch, which is off by default. It has no helper text.

With TLS encryption on, the fields come in this order:

1. An info notice, Certificate chain and trust requirements. It says: If an intermediate CA issued the certificate, include the full chain: the certificate first, then each intermediate. The other end of the connection must trust the issuing CA, so install the CA certificate on machines that do not trust it yet.
2. Certificate.
3. Private key.
4. Passphrase (Optional), with the placeholder Enter the passphrase.
5. The Client certificate verification checkbox. Its helper text says: Enables mutual TLS: every client must present a certificate that verifies against the CA below.
6. While it is checked, CA certificate.
7. Last, Minimum TLS version (Optional) and Maximum TLS version (Optional) in one row. Both show Select a TLS version as the placeholder. The helper text of the minimum says: The lowest TLS version the connection may use. Defaults to TLS 1.2. The helper text of the maximum says: The highest TLS version the connection may use. Leave empty to allow up to TLS 1.3.

The certificate, the private key and the CA certificate each take the text, an environment variable, a Vault secret or a file path, and have an upload link.

Access Settings

The step has the sub-heading Access list. Its description says: Restrict which source networks may reach this listener. Rules are evaluated in order and the first one that matches the sender decides, so put narrower exceptions above broader rules.

• The Source IP filtering switch is Disabled or Enabled. With it enabled, the step shows the Access list editor, with Add rule and the columns Action and IP blocks. While the list has no rule, it says: No rules: every source network may reach this listener.
• A rule has an Action, which is Accept or Drop, and IP blocks. The helper text of the action says: Accepts or drops the senders in this rule's IP blocks. The placeholder of the IP blocks is e.g. 198.51.100.0/24, 192.0.2.10, 203.0.113.1-203.0.113.50. Their helper text says: Separate entries with commas. Each is a CIDR block, a single IP address, a dotted netmask or a low-high range, in IPv4 or IPv6.
• Once the list has a rule, the Unmatched senders field shows. Its choices are Automatic (recommended), Accept and Drop. Its helper text says: The action for a sender that matches no rule. Automatic drops it when the list has an accept rule, and accepts it otherwise.

Advanced Settings

The step has the sub-heading Performance.

• Maximum message size (Optional), a text field. Its placeholder is 33554432. Its helper text says: The largest message a client may send, as it is sent and again once it is decompressed, from 1024 to 1073741824 bytes (1 GiB). A larger message is not accepted. Defaults to 33554432 (32 MiB). On the device page, an empty value reads Default (32 MiB).
• A Tenants section. Its description says: Map source networks to customers on this listener. A message is tagged with the tenant whose IP blocks contain the sender, and when blocks overlap the first listed tenant wins — so put narrower networks above broader ones. It holds the Tenant mapping switch, which is Disabled or Enabled. With it enabled, the section shows the Tenants editor, with Add tenant and the columns Tenant ID and IP blocks. While the list has no tenant, it says: No tenants: no message is tagged with a tenant.
• A tenant has a Tenant ID and IP blocks. The placeholder of the ID is e.g. acme, and its helper text says: The identifier attached to messages that arrive from these networks. The IP blocks have the same placeholder and helper text as in the access list.

Review

The review has one section for each of the three steps:

• The first shows the address (0.0.0.0 when empty) and the port. It also shows the state of Shared key authentication, with the shared key, the username and the password, and the TLS settings. Its row for client certificates is labelled Require Client Authentication.
• The second shows the state of Source IP filtering, each rule in order and the Unmatched senders choice.
• The third shows the maximum message size (33554432 when empty), the state of Tenant mapping and each tenant in order.

It shows the shared key, the password, the certificate and the private key as set or not set, never as themselves.

Validation

The form checks each step with the rules that Director applies to the configuration when the device starts. Director also resolves the secrets, loads the TLS material and binds the port, which the form cannot do, so a device that saves can still fail to start: see Configuration Errors and Errors When the Device Starts. One configuration that the form lets through breaks one of those rules: a device whose YAML has a users list, once you turn Shared key authentication off. The users stay and the shared key goes, and users need a shared key, so the device does not start. In the messages below, <size> stands for a size and <blocks> for the entries that are not valid:

• The address: Listen address must be an IPv4 address, or 0.0.0.0 for every interface.
• The port: Port is required, and Port must be a whole number from 1 to 65535.
• The shared key, with Shared key authentication on: A shared key is required while shared key authentication is on.
• The user: Username is required when Password is set, and Password is required when Username is set.
• With TLS encryption on, the certificate: Certificate is required when it is empty, Provide a valid certificate to continue when the text is not a certificate, This certificate has expired. Provide a valid certificate to continue, and This certificate is larger than <size>. Provide a smaller certificate to continue.
• With TLS encryption on, the key: Private key is required.
• The TLS versions: Maximum TLS version must not be lower than Minimum TLS version (TLS 1.2 when empty).
• Client certificates: Provide a CA certificate to verify client certificates against. It shows when client certificates are required and no CA is given.
• The largest message: Maximum message size must be a whole number of bytes from 1024 to 1073741824.
• A rule of the access list: Action is required, IP blocks must list at least one IP block, and These entries are not valid IP blocks and would be ignored: <blocks>.
• A tenant: Tenant ID is required, IP blocks must list at least one IP block, and the same message about entries that are not valid IP blocks.

The form has no fields for users, self_hostname, timeout, max_connections and max_buffer_size. They are YAML only. A device that has users, self_hostname, timeout or max_connections in its YAML keeps them when you edit the device in the console.

Troubleshooting

Configuration Errors

The Director refuses a configuration it cannot use, with one of these messages. <name> stands for the name of the device, <user> for a user name, <value> for what the field holds, <reason> for the reason the Director gives and N for a number. The log line is Can not validate listener config for <name>. Error: <message>, and the connection state of the device says Configuration validation failed for <name>: <message>:

• port field not valid: the port is outside 1 to 65535. The Director checks the port first, so a port outside the range is reported as a port problem, not as an address problem.
• address not valid: the address cannot be used.
• timeout must not be negative, not N
• max_connections must not be negative, not N
• max_message_bytes must be from 1024 to 1073741824, not N
• users need a shared_key: the handshake that checks them runs only with a shared key, and that holds for a username and a password too
• users is a list of users, each with a username and a password
• every entry of users is a user with a username and a password
• user N of users needs a username and a password
• username is set without a password: a client signs in with both
• password is set without a username: a client signs in with both
• the username "<user>" is also in users: a user is named once
• the shared_key resolves to nothing: clients would sign in with an empty one
• the password of user "<user>" resolves to nothing
• the host name cannot be read: set self_hostname
• tls.status is <value>: it is true or false
• tls.client_auth_required is <value>: it is true or false
• tls.client_ca_name names no CA certificate that loads
• tls.client_ca_name holds no PEM certificate: it must be a CA certificate in PEM form
• failed to build TLS config: <reason>: the TLS settings cannot be used. <reason> says why.
• tls: the certificate or the private key does not load
• a secret could not be resolved: the <what> of device "<name>": <reason>: a variable that is not set, or a vault that does not answer. <what> is shared_key, password of user "<user>", TLS certificate, TLS private key or TLS clients' CA. A secret that cannot be read just now does not restart a running device.

Errors When the Device Starts

These appear in the Director's log.

• Can not start FluentForward listener for <name>. Error: <reason>: the port check failed, for example because the port is not free. The device does not start. The connection state of the device says Connection validation failed for <name>: <reason>.
• Can not initialize writers for <name>. Error: <reason>: the device can take nothing, so the Director stops it.
• Failed to get Fluent Forward server. Error: <reason>: the device can take nothing, so the Director stops it.
• Failed to run Fluent Forward server. Error: <reason>: the server stopped with an error. The Director also reports the failure in the connection state of the device.
• Skipping initialization of listener.: the device or the Director service is turned off, so the Director does not start it. It is an information line.

When the port check passes, the connection state of the device says FluentForward listener ready for <name>.

Log Lines While It Runs

These appear in the Director's log. <name> stands for the name of the device, <client> for the address and port that the connection came from, and N for a count. The device warns at most once a minute for each of three kinds: clients refused at the handshake, dropped events or messages, and messages with metrics or traces.

Information:

• Listener (<name>) server is successfully started.

Warnings:

• A Fluent Forward client at <client> was refused: shared_key mismatch. The key of the client is wrong.
• A Fluent Forward client at <client> was refused: username/password mismatch. The user name is unknown, or the password is wrong.
• A Fluent Forward client at <client> was refused: its first message is not a PING. The client sent a message in place of a PING, so it does not run the handshake.
• The Fluent Forward device takes logs only. It dropped a message with metrics or traces from <client>.
• Fluent Forward dropped an event from <client> that is larger than the ingest buffer can take.
• Fluent Forward dropped an event from <client> that the ingest buffer can never take: the record can never be taken: <reason>.
• Fluent Forward dropped a message from <client> and closed its connection: <reason>. The reason is fluent forward: message too large, or fluent forward: malformed message: <detail>. The details are:
  • a message holds 2 to 4 values, not N
  • the tag is not a string
  • the tag is N bytes long, and at most N are allowed
  • a message with a time holds a record too
  • the second value of the message is neither a time nor a list of events
  • an event holds a time and a record, not N values
  • the option is not a map
  • the chunk is not a string
  • the chunk is N bytes long
  • compressed is not a string
  • the compression "<name>" is not supported
  • gzip: <reason>
  • an event is cut short
  • nested deeper than 64 levels
  • the event time <value> is out of range
  • an extension of type N with N bytes is not an event time
  • an event time cannot start with code <hex>
  • a value cannot start with code <hex>
  • nil where a string belongs
  • a record is a map, not a value with code <hex>
  • another reason why the message cannot be read

Errors:

• Can not encode incoming message for <name>. Error: <reason>
• Can not write incoming events for <name> to disk; the client's connection is closed, and it sends them again. Error: <reason>
• Fluent Forward receiver for <name> dropped N message(s) or event(s) that it could not take. The Director logs it at each statistics interval, when something was dropped.
• Failed to update device configuration for <name>
• FluentForward heartbeat check failed. Stopping collector...: the device stopped answering its health check, and the Director stops it. The connection state of the device says FluentForward heartbeat check failed for <name>.

Verbose:

• Fluent Forward receiver for <name> handled N connection(s): received N event(s), wrote N, dropped N, and refused N client(s) at the handshake. The Director logs it at each statistics interval in which a count changed. The numbers cover that interval.

Debug:

• Can not resolve listener config for <name>. Error: <reason>: an edit no longer resolves. The device starts again, and the start reports why it cannot run. A secret that cannot be read just now is the exception: the device keeps running.

What the Client Sees

• A refused handshake: the device answers PONG with false and the reason shared_key mismatch, username/password mismatch or that the message is not a PING. Then it closes the connection.
• A message that is malformed or too large: no answer. The device stores and acknowledges what the client sent before it, then closes the connection.
• Events that the Director cannot store: no answer, and the connection closes. The sender sends the message again.
• A record that is too large for the ingest buffer: the device drops the record and still acknowledges the message.
• A message with metrics or traces: the device drops it and acknowledges it when the sender asked for that.
• Fluentd marks the Director as down: its heartbeat_type is udp. The device answers no UDP heartbeat. Use transport or none.

Examples

Basic

Fluent Bit and Fluentd send their events to port 24224, with no handshake and no TLS. Use this only on a network where nobody else can read the traffic...

- id: 1 name: fluent-forward description: Fluent Bit and Fluentd sending their logs over the forward protocol type: fluent_forward properties: port: 24224

Shared Key, a User and TLS

The same with the handshake, one user and TLS. A client must know the shared key and sign in as fluent-bit. To check the certificates of the clients (mTLS), name the CA that signed them; client_auth_required refuses a client that presents none...

- id: 2 name: fluent-forward-secure description: Fluent Bit and Fluentd signing in with a shared key and a user, over TLS type: fluent_forward properties: shared_key: "${FLUENT_FORWARD_SHARED_KEY}" self_hostname: "director-1" users: - username: "fluent-bit" password: "${FLUENT_FORWARD_PASSWORD}" tls: status: true cert_name: "fluent.example.com.crt" key_name: "fluent.example.com.key" min_tls_version: "tls1.2" # client_ca_name: "clients-ca.crt" # client_auth_required: true

Limits, Access List and Tenants

Shorter waits and smaller messages. The access list lets in only the 10.0.0.0/8 network, and a message from 10.1.0.0/16 carries the tenant acme...

- id: 3 name: fluent-forward-limits description: Fluent Bit and Fluentd sending their logs from the 10.0.0.0/8 network type: fluent_forward properties: timeout: 120 max_connections: 1000 max_message_bytes: 8MB access_list: - action: accept ip_blocks: "10.0.0.0/8" tenants: - tenant_id: "acme" ip_blocks: "10.1.0.0/16"