ClickHouse

Databases Analytics

Synopsis

Creates a ClickHouse target that sends log data to a ClickHouse database server for analytics and storage. Supports batch processing and field normalization.

Schema

- name: <string>
  description: <string>
  type: clickhouse
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <integer>
    protocol: <string>
    format: <string>
    secure: <boolean>
    insecure_skip_verify: <boolean>
    tls:
      ca_name: <string>
      server_name: <string>
      min_tls_version: <string>
      max_tls_version: <string>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
    username: <string>
    password: <string>
    database: <string>
    table: <string>
    message_column: <string>
    compression: <string>
    max_open_conns: <integer>
    block_buffer_size: <integer>
    max_bytes: <integer>
    field_format: <string>
    timeout: <integer>
    async_insert: <boolean>
    async_insert_wait: <boolean>
    batch_size: <integer>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be clickhouse
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
address	Y	-	ClickHouse server address. A bare host name or IP address, without a scheme and without a port
port	N	8123 on http, 8443 with secure: true; 9000 on tcp, 9440 with secure: true	Port the server listens on. The default follows the transport and secure together, so a server on the standard ports needs no port at all. A port you set yourself is always used as written
protocol	N	http	Transport. http inserts over the HTTP interface, default port 8123, or 8443 with secure: true; tcp, alias native, uses ClickHouse's native protocol, default port 9000, or 9440 with secure: true. TLS is secure, on either transport
format	N	jsoneachrow, or native on tcp	Encoding carried over the transport. jsoneachrow, tabseparatedraw and parquet are the HTTP encodings, native the one the TCP protocol carries. See Protocol and Format below
secure	N	false	Connect over TLS. Must match the port the server serves TLS on
insecure_skip_verify	N	false	With secure: true, accept the server certificate without verifying it. Use only for testing. Top-level under properties, not under tls
tls.ca_name	N	-	With secure: true, the CA certificate that signed the server's certificate. When unset, the host trust store is used; when set, it replaces the host trust store rather than adding to it
tls.server_name	N	-	With secure: true, the name the server's certificate is verified for, also sent as the SNI. When unset, the host the Director connects to
tls.min_tls_version	N	tls1.2	With secure: true, the lowest TLS version the connection may use: tls1.0, tls1.1, tls1.2, tls1.3
tls.max_tls_version	N	tls1.3	With secure: true, the highest TLS version the connection may use. Must not be lower than tls.min_tls_version
tls.cert_name	N	-	With secure: true, the client certificate the Director presents for mutual TLS. Requires tls.key_name
tls.key_name	N	-	The private key matching tls.cert_name. Requires tls.cert_name
tls.passphrase	N	-	Decrypts tls.key_name when the key is encrypted
username	N	-	ClickHouse user name. Left empty, the server's default user is used
password	N	-	Password for that user
database	Y	-	ClickHouse database name
table	Y	-	ClickHouse table name
max_bytes	N	67108864 (64 MB)	Maximum size of a single request body or batch, in bytes
timeout	N	30	Seconds allowed for a single insert request, including the response. The http transport only

note

max_bytes caps the size of a single request body or batch. It behaves differently from max_size on the file-writing targets, and the difference is easy to get wrong:

• An explicit max_bytes: 0 IS honoured and removes the ceiling on targets that have no ceiling of their own. The value is read with Int64, which returns what you wrote — unlike max_size, which is read with MustInt64 and silently substitutes its default for a zero. Where a target has a fixed ceiling of its own, its page says so and a zero is clamped to that ceiling instead.
• A negative value is rejected at configuration time with max_bytes must not be negative.

The ceiling is validated when the target is built and enforced again on every batch, so a value the receiving service will refuse fails early rather than per request.

note

By default the target inserts over ClickHouse's HTTP interface, so port must be an HTTP port of the server: 8123, or 8443 with secure: true. Set protocol: tcp to insert over the native protocol instead, where the port is 9000, or 9440 with secure: true. Left unset, port resolves to whichever of those four matches the transport and secure. Writing format: native on its own selects the same transport as protocol: tcp, and stays supported.

note

If you are upgrading: a target that names a port, or that runs without TLS, keeps the port it has always used. Only a target that sets secure: true and leaves port unset resolves differently now, and such a target could not connect before, because it dialled the plaintext port with TLS.

Protocol and Format

Two keys on two axes. protocol picks the transport, and with it the port default. format picks the encoding carried over that transport. Values on both keys are matched case-insensitively, and each accepts aliases.

protocol	Aliases	Default port	Encoding it carries
http	-	8123, or 8443 with secure: true	Any one of the three HTTP formats below
tcp	native	9000, or 9440 with secure: true	native, always

format	Aliases	Transport	What is sent
jsoneachrow	json, jsoneach	http	One JSON object per record, a column per field. The insert asks the server to skip fields the table does not declare
tabseparatedraw	raw, tsv, tabseparated	http	The raw message text, one row per record, into a single column. Tabs, newlines and backslashes in the message are escaped, and the server restores the original bytes
parquet	-	http	A Parquet block holding the raw message text in a single column
native	tcp	tcp	The raw message text in ClickHouse's native binary blocks

Only two pairings exist on the wire, because the native protocol carries its own block format and no other encoding is reachable over it:

http + (jsoneachrow | tabseparatedraw | parquet)      tcp + native

Any other combination is refused when the configuration is loaded, so you find out at reload rather than watching the target start and fail on every flush.

What a configuration resolves to

Configuration	Resolves to
Neither key	http + jsoneachrow
format: native	tcp + native. The transport is inferred from the encoding, which is how this was written before protocol existed. Still supported
format: tcp	tcp + native, by the same inference
protocol: tcp	tcp + native. The format default here is native, not jsoneachrow
protocol: native	tcp + native. On this key native is an alias of tcp
protocol: http with any HTTP format	http + that format
protocol: tcp with any HTTP format	Refused
protocol: http with format: native	Refused
protocol: https	Refused. TLS is the secure flag, on either transport

An absent key is not the same as one set to its default: only an absent protocol is inferred from the format, and only an absent format is filled in by the transport.

Field	Required	Default	Description
message_column	N	message	Column that receives the raw message text under tabseparatedraw, parquet and native. Unused by jsoneachrow, which writes a column per field
compression	N	gzip	Gzip the request body. none disables it, as does a legacy use_compression: false. Applies to jsoneachrow and tabseparatedraw only, because Parquet bodies carry their own compression and the native protocol does not use it
max_open_conns	N	The CPU count of the host, bounded to between 16 and 64	protocol: tcp only. Size of the TCP connection pool the sender workers share
block_buffer_size	N	16	protocol: tcp only. Data blocks the native protocol pipelines per insert. Values outside 1 to 255 are clamped

note

In the web interface the same choice appears as the Protocol and Format controls of the target's Connection settings step, beside a TLS encryption toggle. The port the step pre-fills follows both the protocol and that toggle, so it changes as you switch either one. Turning the toggle on shows the TLS fields: Skip certificate verification, CA certificate, Server name, Minimum TLS version and Maximum TLS version, and an optional client Certificate, Private key and Passphrase. For a server whose certificate is publicly trusted, the toggle alone is enough; for a private certificate authority, paste or upload its certificate in CA certificate. The console omits a blank field, and all of them while TLS encryption is off.

Processing

Field	Required	Default	Description
field_format	N	-	Data normalization format. See applicable Normalization section
async_insert	N	false	Let the server buffer small inserts and batch them itself. The http transport only
async_insert_wait	N	false	With async_insert, wait for the rows to be written before the insert is acknowledged. Left off, the insert is fire-and-forget
batch_size	N	25000, or the service-level batch_size when set	Records per insert. Overrides the service-level batch_size for this target. Must be a whole number of at least 1; a blank value counts as unset

note

The target's own batch_size sets the records per insert. Left unset, the target falls back to the service-level batch_size in vmetric.yml, and then to 25000, not the generic default of 1000, because each flush costs a round-trip to the server. The value is not capped at 10000 as on the HTTP-batching targets; max_bytes still bounds each batch.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The ClickHouse target inserts log data in batches. By default it inserts over ClickHouse's HTTP interface, on port 8123, or on the HTTPS port when secure is enabled. With protocol: tcp it inserts over ClickHouse's native protocol instead, on port 9000 by default. Logs are accumulated until the batch size is reached, then sent to the server. The batch size is the target's own batch_size, or else the service-level batch_size, or else 25000.

The target can normalize field names before sending data to ClickHouse, and field_format accepts any format the Normalize processor supports, not just a subset. Which one to pick depends on what reads the table. ClickHouse's own observability stack, ClickStack, stores logs in the OpenTelemetry logs schema by default, so otel is the usual choice when the data is queried that way.

Prerequisites

1. A running ClickHouse server
2. Its HTTP port, or its native TCP port with protocol: tcp, reachable from the Director host
3. A database and table already created in ClickHouse
4. A user with write permissions to the specified table

note

The table has to match what the format sends. Under jsoneachrow that means a column per field the records carry, which for a raw log line is a message column of type String and an @timestamp column. Under tabseparatedraw, parquet and native it means a single String column named by message_column, because those formats send the message text and nothing else.

warning

For high-volume logging, ensure your ClickHouse server is properly configured for performance, including appropriate settings for inserts, memory usage, and disk I/O.

Examples

Basic

Minimum configuration for sending logs to ClickHouse:

targets:
  - name: basic_clickhouse
    type: clickhouse
    properties:
      address: "192.168.1.100"
      username: "default"
      password: "password"
      database: "logs"
      table: "system_logs"

Custom Port

Configuration with a non-default port, here the HTTPS port with TLS enabled:

targets:
  - name: custom_port_clickhouse
    type: clickhouse
    properties:
      address: "clickhouse.example.com"
      port: 8443
      secure: true
      username: "logger"
      password: "secure_password"
      database: "logs"
      table: "application_logs"

Native Protocol

Inserting over ClickHouse's native protocol, which is the faster path for high-volume ingest:

targets:
  - name: native_clickhouse
    type: clickhouse
    properties:
      address: "clickhouse.example.com"
      protocol: "tcp"
      username: "logger"
      password: "secure_password"
      database: "logs"
      table: "high_volume_logs"
      message_column: "message"

port is left out, so it defaults to 9000, and format is left out because the TCP protocol carries only its own native encoding. The table needs a single String column named by message_column, because this path writes the raw message text rather than a column per field.

Older configurations spell the same target as format: "native" with no protocol. That still resolves to the same transport and encoding, and needs no change.

With Normalization

Configuration using field normalization:

targets:
  - name: normalized_clickhouse
    type: clickhouse
    properties:
      address: "clickhouse.example.com"
      username: "logger"
      password: "secure_password"
      database: "logs"
      table: "security_logs"
      field_format: "ecs"

With Pipeline

Using a pipeline for additional log processing:

targets:
  - name: pipeline_clickhouse
    type: clickhouse
    pipelines:
      - enrich_logs
    properties:
      address: "clickhouse.example.com"
      username: "logger"
      password: "secure_password"
      database: "logs"
      table: "enriched_logs"

Secure Configuration

Using environment variables for credentials:

targets:
  - name: secure_clickhouse
    type: clickhouse
    properties:
      address: "${CLICKHOUSE_ADDRESS}"
      username: "${CLICKHOUSE_USER}"
      password: "${CLICKHOUSE_PASSWORD}"
      database: "logs"
      table: "security_logs"

Troubleshooting

This section covers the errors you are most likely to see with the clickhouse target, what causes each one, and how to fix it.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line.

See Target Delivery Errors for how Director logs and retries target failures.

note

The target contacts the server only when it sends a batch. A wrong port, a wrong password, a missing grant and a missing table all surface on the first flush rather than at startup, so the connection status can look healthy until data first arrives.

Which permission or setting is missing?

On the http transport, a refused insert comes back as an HTTP status followed by a ClickHouse exception code. The code is the part you act on. Match it against this table first.

A protocol: tcp target reports the same exception codes without an HTTP status, wrapped as clickhouse native prepare, clickhouse native append or clickhouse native send. Read the code out of the message and use the same table: the HTTP 4xx column is what the HTTP interface puts in front of it, not part of the diagnosis.

Error text	What is missing	Where to fix it
HTTP 403 with Code: 516, Code: 192, Code: 193 or Code: 194	A user whose name and password match username and password	The user definition on the ClickHouse server
HTTP 403 with Code: 195, typically	A user whose HOST clause accepts connections from the Director host	The user definition on the ClickHouse server
HTTP 403 with Code: 497, typically	Insert permission on the table, for example GRANT INSERT ON logs.system_logs TO logger	The grants of the user in username
HTTP 404 with Code: 81, typically	The database named in database. The target never creates one	The ClickHouse server
HTTP 404 with Code: 60, typically	The table named in table. The target never creates one	The ClickHouse server
HTTP 400 with Code: 27, or Code: 16 typically	Columns that match the records you send	The table definition
Code: 164, typically	A user profile that allows per-query settings, because each insert sets input format options	The profile of the user in username
connection refused, or malformed HTTP response	A reachable port for the transport in use	port, protocol, secure, and the firewall between Director and the server

"connection refused", "malformed HTTP response", or a batch rejected on the native port

[Error] [director] [target-<target id>] [basic_clickhouse] Sender worker 3 Finalize failed on flush for target "basic_clickhouse": clickhouse insert request: Post "http://clickhouse.example.com:9000/?database=logs&query=INSERT+INTO+system_logs+FORMAT+JSONEachRow": dial tcp ...: connect: connection refused

Cause: port is not an HTTP port of the server, on a target whose transport is http. That transport inserts over the ClickHouse HTTP interface, so a port that serves the native protocol cannot answer it. The symptom depends on what is listening.

On a target whose transport is tcp this diagnosis does not apply. There the port to reach is 9000, or 9440 when secure: true is set, and a failure on the right one means the server is unreachable rather than that the port is wrong. The transport is whatever protocol says where it is set, and is inferred from format: native where it is not.

Error text	What it means
connect: connection refused	Nothing listens on that port, or a firewall rejects the connection
malformed HTTP response "\x15\x03\x01..."	A plain request reached a TLS port. Add secure: true, or use the plain HTTP port
http: server gave HTTP response to HTTPS client	secure: true was used against the plain HTTP port. Remove secure, or use the HTTPS port
HTTP 400 with no Code: prefix	An HTTP request reached the native protocol port, which typically answers with a plain 400 telling you to use the HTTP port instead. If reaching that port was the intention, set protocol: tcp

Fix: leave port unset unless your server listens somewhere other than the standard ports. It then resolves to 8123, or 8443 with secure: true, and under protocol: tcp to 9000, or 9440 with secure: true. Where you do set a port, make sure it matches secure, because the port you write is always the one dialled. Keep address a bare host name or IP address. A value such as http://clickhouse.example.com is treated as a host name and fails with dial tcp: lookup http: no such host.

Connection failures are retried until you fix them, and nothing is lost.

warning

The 400 answer from a native protocol port is the exception. Any 400 counts as a permanent rejection, so that batch is attempted four times and then given up on. Data sent to the wrong port this way is lost, not queued.

"HTTP 403" with "Code: 516"

[Error] [director] [target-<target id>] [basic_clickhouse] Sender worker 2 Finalize failed on flush for target "basic_clickhouse": clickhouse insert failed: HTTP 403: Code: 516. DB::Exception: ... (AUTHENTICATION_FAILED)

Cause: the server refused the credentials. The text after DB::Exception: typically says that the password is incorrect or that no user of that name exists. Code: 192, Code: 193, Code: 194 and Code: 195 separate the same failure into unknown user, wrong password, password required, and host not allowed for this user.

Fix: check username and password against the user on the server, and watch for whitespace pasted along with either value. A ${VAR} or $secret{...} reference that resolves to an empty string leaves username empty, and an empty username connects as the ClickHouse default user instead, which is rarely what you intended. If the credentials are right, check that the HOST clause of the user accepts the Director host.

Nothing is lost. The batch stays queued and is retried every few seconds until the server accepts it.

"HTTP 403" with "Code: 497"

Sender worker 5 Finalize failed on flush for target "basic_clickhouse": clickhouse insert failed: HTTP 403: Code: 497. DB::Exception: ... (ACCESS_DENIED)

Cause: the user authenticated, but it has no insert permission on the table. The message typically names the grant the server wanted, down to the individual columns.

Fix: grant insert on the exact database and table the target writes to, then wait for the next retry:

GRANT INSERT ON logs.system_logs TO logger

Nothing is lost. The batch is retried until the grant is in place.

"HTTP 404" with "Code: 60" or "Code: 81"

Sender worker 1 Finalize failed on flush for target "basic_clickhouse": clickhouse insert failed: HTTP 404: Code: 60. DB::Exception: ... (UNKNOWN_TABLE)

Cause: the target never creates databases or tables. Code: 60 typically means the table named in table does not exist, and Code: 81 that the database named in database does not exist. A typo in either field looks exactly like an object that was never created.

Fix: create the objects, or correct database and table. Keep the database in database only. A value such as logs.system_logs in table is rejected before the target starts, because both fields accept letters, digits and underscores only. The table also needs columns that match what the format sends: under jsoneachrow a column per emitted field, which for raw lines means a message column of type String and an @timestamp column, and under the other three formats the single column named by message_column.

Nothing is lost. The batch is retried until the object exists.

"record rejected by target" with "Code: 27"

Sender worker 6 Finalize failed on flush for target "basic_clickhouse": record rejected by target: clickhouse insert failed: HTTP 400: Code: 27. DB::ParsingException: ... (CANNOT_PARSE_INPUT_ASSERTION_FAILED)

Sender worker 6 dropping natsobj:/vm-pipeline-payloads/sender.basic_clickhouse.<id> for target "basic_clickhouse" after 4 rejected flush attempts — the endpoint refuses this batch deterministically, so its records are given up on

Cause: at least one row in the batch does not fit the table. A string arrived in a numeric column, a date could not be parsed, a null arrived in a column that is not Nullable, or a record was not valid JSON. Code: 41, Code: 117 and Code: 349 typically point at the same class of problem, and Code: 16 means the column the batch writes to is not in the table at all.

warning

This is the error on this page that loses data. The rejection is permanent, so the batch is attempted four times and then given up on. One bad row takes the whole batch with it, and a batch holds up to batch_size records (the target's, or else the service's), or max_bytes of data.

Fix: align the table with the data. Add the columns your records carry, make a column Nullable where the value can be absent, and keep a String column for the message when the pipeline emits raw lines. The message typically quotes the text the server choked on and the row number inside the batch, which tells you which record to correct in the pipeline that feeds the target.

"HTTP 413", or "record size ... exceeds configured max_bytes"

Sender worker 3 Finalize failed on flush for target "basic_clickhouse": record rejected by target: clickhouse insert failed: HTTP 413: ...

Sender worker 3 execute() failed for basic_clickhouse: record rejected by target: record size 1048612 exceeds configured max_bytes 1048576

Cause: a 413 comes from a proxy or load balancer in front of ClickHouse whose body limit is smaller than max_bytes, which defaults to 64 MB. ClickHouse itself publishes no limit on the size of an insert body. The second message is a different case: one single record is larger than max_bytes, which only happens once you set max_bytes below the size of your largest event.

Fix: for the 413, lower max_bytes under the limit of the proxy, or raise the limit of the proxy. For the oversized record, raise max_bytes, or trim the event in a pipeline before it reaches the target. max_bytes is counted on the message text, so the request body on the wire is somewhat larger than the number you set.

Both are permanent rejections. The 413 gives up on the whole batch after four attempts. The oversized record is dropped alone on its fourth attempt, logged as Target "<name>" rejected record <n> of <payload> on its last delivery attempt — dropping that record and delivering the rest of the payload: <reason> and counted on the target's dropped counter, and the rest of its payload is delivered. Those records are lost.

"HTTP 429" with "Code: 202", or an "HTTP 500"

Sender worker 7 Finalize failed on flush for target "basic_clickhouse": clickhouse insert failed: HTTP 429: Code: 202. DB::Exception: ... (TOO_MANY_SIMULTANEOUS_QUERIES)

Cause: the server is refusing the work, not the data. Code: 202 means the number of queries running at once has reached the limit of the server. The 500 answers are the same pressure from another angle: Code: 241 typically for a batch too large for the memory of the server, Code: 252 typically when merges cannot keep up with the rate of inserts, and Code: 242 or Code: 243 typically for a table in read-only mode or a disk with no space left.

Fix: send fewer and larger inserts, and give the server room. Raise the limit on concurrent queries, reduce the number of sender workers, and lower max_bytes when the server runs out of memory. For Code: 252, flush on a schedule with interval or cron as described under Scheduling and Pool Behavior, so that fewer and bigger inserts arrive.

Nothing is lost. Throttling and server errors are retried until the server accepts the batch.

"no such host", "i/o timeout", or "certificate signed by unknown authority"

Sender worker 0 Finalize failed on flush for target "custom_port_clickhouse": clickhouse insert request: Post "https://clickhouse.example.com:8443/?database=logs&...": tls: failed to verify certificate: x509: certificate signed by unknown authority

Cause: Director cannot reach the server, or cannot trust it. The text after the URL says which step failed.

• dial tcp: lookup clickhouse.example.com: no such host means the name does not resolve on the Director host. A scheme left in address appears here as lookup http: no such host.
• i/o timeout, or context deadline exceeded (Client.Timeout exceeded while awaiting headers), means the traffic is dropped on the way, or the insert took longer than timeout, which defaults to 30 seconds. A large batch on a slow link fails this way on every attempt.
• x509: certificate signed by unknown authority means the server presents a certificate from a private CA. Point tls.ca_name at that CA certificate, or, in the web interface, paste or upload it in CA certificate.
• x509: certificate is valid for ..., not ... means you connected by IP address, or under a name the certificate does not cover. Use the name in the certificate, or set tls.server_name.
• proxyconnect tcp: ... means HTTP_PROXY or HTTPS_PROXY is set for the Director service and the proxy cannot be reached. Add the ClickHouse host to NO_PROXY.

Fix: correct address, open the port, or supply the CA certificate. For a slow link, raise timeout or lower max_bytes so that each request is smaller. Nothing is lost while this lasts, and the batch is retried until it succeeds.

"address is required", and other configuration errors

These stop the target from starting. They appear as ValidateConfig failed for target "basic_clickhouse": ..., or as Failed to reinitialize target "basic_clickhouse" (attempt N). Reason: ... for the TLS files, and are re-checked until you correct the configuration. Nothing is sent in the meantime, and incoming data waits in the queue.

Reason text	Fix
address is required for clickhouse target	Set address to a bare host name or IP address
database is required for clickhouse target	Set database
table is required for clickhouse target	Set table
invalid database name "..." for clickhouse target	Names take letters, digits and underscores, and cannot start with a digit. Rename the database
invalid table name "..." for clickhouse target	The same rule. logs.system_logs is not accepted here: put the database in database and the bare name in table
invalid format "..." for clickhouse target (valid: jsoneachrow, tabseparatedraw, parquet, native)	Use one of those four values or an accepted alias. An empty value is jsoneachrow
invalid protocol "..." for clickhouse target (valid: http, tcp); for TLS keep the protocol and set secure: true	Use http or tcp, or native as an alias of tcp. https is not a value here: keep the transport and set secure: true
format "..." is not valid with protocol: tcp for clickhouse target ...	The native protocol carries only its own block format. Use protocol: http for that encoding, or drop the format key and let the transport fill it in
format: native is not valid with protocol: http for clickhouse target ...	native is the block format of the TCP protocol. Use protocol: tcp, or pick jsoneachrow, tabseparatedraw or parquet
invalid message_column "..." for clickhouse target	The same rule as the database and table names. Rename the column
clickhouse target: max_bytes must not be negative, got ...	Use a positive value, or 0 to remove the ceiling
batch_size must be a whole number greater than 0 for clickhouse target, got ...	Use a whole number of at least 1, or leave batch_size blank to use the service-level value
failed to build TLS config: ...	The rest of the line names the option at fault: a certificate path that is not under the service root, a file that holds no PEM certificate, an unresolvable ${VAR}, a certificate without its key, or a wrong passphrase

The target is healthy but nothing arrives in ClickHouse

Nothing fails here, so there is no error to search for. Check the following in order.

1. debug.dont_send_logs is enabled. Records are processed by the pipeline and then discarded before anything is buffered. No counter moves and no insert is made, while the target reports healthy. The only trace is one line at startup, and only when debug.status is enabled as well:

   Log sending is disabled for this target (basic_clickhouse). Logs are processed but not sent.

2. The table does not declare the fields you send. Under jsoneachrow, each record is inserted as JSON and the insert asks the server to skip fields the table does not know. A record whose keys match no column typically becomes a row of column defaults rather than an error, and the target counts it as delivered. Name the columns after the keys your pipeline emits, and query the table for recent rows before assuming the data never arrived. The other three formats do not have this failure mode: they write the raw message text into message_column, and a missing column is an error rather than a silent default.

3. Asynchronous inserts are acknowledged before they are written. On the http transport, with async_insert: true and without async_insert_wait: true, the server acknowledges the insert as soon as it is buffered. Rows that are then lost from that buffer are never reported back. Set async_insert_wait: true, or leave async_insert off.

4. A batch was given up on. A dropped batch moves no counter of its own. Search the logs for after 4 rejected flush attempts and for dropping (giving up), and see the parsing and size entries above for the causes.

5. Normalization failed quietly. A record whose field_format conversion fails is still sent as it stands. If the result is not valid JSON it can fail the parsing check for the whole batch, so a silent normalization problem surfaces as the drop described above.

6. The record had an empty body. A normalized record with no message body is skipped by the encoder but still counted as delivered, so the counts can exceed the rows in the table. Drop empty records in a pipeline if the difference matters.