Redpanda

Redpanda Message Queue

Synopsis

Creates a target that writes log messages to Redpanda topics with support for batching, compression, and authentication. The target handles message delivery efficiently with configurable batch limits based on size or event count. Redpanda is a Kafka-compatible streaming platform that provides API compatibility with Apache Kafka.

Schema

- name: <string>
  description: <string>
  type: redpanda
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <numeric>
    client_id: <string>
    topic: <string>
    algorithm: <string>
    username: <string>
    password: <string>
    compression: <string>
    compression_level: <string>
    acknowledgments: <string>
    allow_auto_topic_creation: <boolean>
    disable_idempotent_write: <boolean>
    max_bytes: <numeric>
    max_events: <numeric>
    field_format: <string>
    tls:
      status: <boolean>
      insecure_skip_verify: <boolean>
      min_tls_version: <string>
      max_tls_version: <string>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
    interval: <string|numeric>
    cron: <string>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be redpanda
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Redpanda Connection

Field	Required	Default	Description
address	Y	-	Redpanda broker address (hostname or IP)
port	N	9092	Redpanda broker port (9092 for plaintext, 9093 for TLS)
client_id	N	-	Client identifier for connection tracking
topic	Y	-	Topic name for message delivery

Authentication

Field	Required	Default	Description
algorithm	N	"none"	Authentication mechanism: none, plain, scram-sha-256, scram-sha-512
username	N*	-	Username for SASL authentication
password	N*	-	Password for SASL authentication

* = Conditionally required when using SASL authentication.

Producer Settings

Field	Required	Default	Description
compression	N	"none"	Message compression: none, gzip, snappy, lz4, zstd
compression_level	N	-	Compression level (algorithm-specific)
acknowledgments	N	"leader"	Acknowledgment level: none, leader, all
allow_auto_topic_creation	N	false	Allow automatic topic creation if topic doesn't exist
disable_idempotent_write	N	false	Disable idempotent producer (not recommended)

Batch Configuration

Field	Required	Default	Description
max_bytes	N	0	Maximum batch size in bytes (0 = unlimited)
max_events	N	1000	Maximum number of events per batch
field_format	N	-	Data normalization format. See applicable Normalization section

note

Batches are sent when either max_bytes or max_events limit is reached, whichever comes first.

TLS Configuration

Field	Required	Default	Description
tls.status	N	false	Enable TLS/SSL encryption
tls.insecure_skip_verify	N	false	Skip certificate verification (not recommended)
tls.min_tls_version	N	"tls1.2"	Minimum TLS version: tls1.2, tls1.3
tls.max_tls_version	N	"tls1.3"	Maximum TLS version: tls1.2, tls1.3
tls.cert_name	N	"cert.pem"	Client certificate file name for mTLS
tls.key_name	N	"key.pem"	Private key file name for mTLS
tls.passphrase	N	-	Passphrase for encrypted private key

Scheduler

Field	Required	Default	Description
interval	N	realtime	Execution frequency. See Interval for details
cron	N	-	Cron expression for scheduled execution. See Cron for details

Details

Redpanda is a Kafka-compatible streaming platform. This target type allows you to connect to Redpanda clusters using the standard Kafka protocol.

Kafka API Compatibility

Redpanda is fully compatible with Apache Kafka APIs. You can use standard Kafka client libraries without modification. All Kafka protocol features supported by the DataStream Kafka target work with Redpanda.

Authentication Methods

Redpanda supports multiple SASL authentication mechanisms:

• No Authentication: Open access without credentials
• SASL/PLAIN: Simple username/password authentication
• SASL/SCRAM-SHA-256: Secure challenge-response authentication
• SASL/SCRAM-SHA-512: Enhanced secure authentication

Connection Requirements

• Default port: 9092 (plaintext) or 9093 (TLS)
• Bootstrap server format: hostname or IP address
• For Redpanda Cloud: Use the bootstrap server provided in the cluster details

TLS Configuration

Redpanda supports both plaintext and TLS-encrypted connections:

• Plaintext: Port 9092 (default)
• TLS: Port 9093 (enable with tls.status: true)

For production deployments, TLS encryption is recommended.

Message Delivery Guarantees

The acknowledgments setting controls delivery guarantees:

Level	Behavior	Use Case
none	No acknowledgment from broker	Maximum throughput, lowest durability
leader	Acknowledgment from partition leader only	Balanced throughput and durability
all	Acknowledgment from all in-sync replicas	Maximum durability

Compression

Message compression reduces network bandwidth and storage requirements:

Algorithm	Compression Ratio	CPU Usage	Speed
none	None	Minimal	Fastest
gzip	High	High	Slow
snappy	Medium	Low	Fast
lz4	Medium	Low	Fast
zstd	High	Medium	Medium

Examples

Basic Configuration

The minimum configuration for a Redpanda target:

targets:
  - name: basic_redpanda
    type: redpanda
    properties:
      address: "redpanda.example.com"
      topic: "application-logs"

With SASL/SCRAM-SHA-256

Configuration using SCRAM-SHA-256 authentication:

targets:
  - name: secure_redpanda
    type: redpanda
    properties:
      address: "redpanda.example.com"
      port: 9093
      topic: "secure-logs"
      algorithm: "scram-sha-256"
      username: "log-producer"
      password: "secure-password-here"
      tls:
        status: true
        insecure_skip_verify: false

With SASL/SCRAM-SHA-512

Configuration using SCRAM-SHA-512 authentication:

targets:
  - name: redpanda_scram512
    type: redpanda
    properties:
      address: "redpanda.example.com"
      port: 9093
      topic: "authenticated-logs"
      algorithm: "scram-sha-512"
      username: "kafka-user"
      password: "strong-password"
      tls:
        status: true

With TLS Encryption

Configuration with TLS enabled:

targets:
  - name: encrypted_redpanda
    type: redpanda
    properties:
      address: "redpanda.example.com"
      port: 9093
      topic: "encrypted-logs"
      tls:
        status: true
        insecure_skip_verify: false
        min_tls_version: "tls1.2"
        max_tls_version: "tls1.3"

With mTLS Authentication

Configuration using mutual TLS authentication:

targets:
  - name: mtls_redpanda
    type: redpanda
    properties:
      address: "redpanda.example.com"
      port: 9093
      topic: "mtls-logs"
      tls:
        status: true
        insecure_skip_verify: false
        min_tls_version: "tls1.2"
        cert_name: "client-cert.pem"
        key_name: "client-key.pem"
        passphrase: "key-passphrase"

With Compression

Configuration with lz4 compression:

targets:
  - name: compressed_redpanda
    type: redpanda
    properties:
      address: "redpanda.example.com"
      topic: "compressed-logs"
      compression: "lz4"

High Throughput

Configuration optimized for maximum throughput:

targets:
  - name: high_throughput_redpanda
    type: redpanda
    properties:
      address: "redpanda.example.com"
      topic: "high-volume-logs"
      compression: "lz4"
      acknowledgments: "leader"
      max_bytes: 1048576
      max_events: 10000

High Reliability

Configuration optimized for maximum durability:

targets:
  - name: reliable_redpanda
    type: redpanda
    pipelines:
      - checkpoint
    properties:
      address: "redpanda.example.com"
      topic: "critical-logs"
      compression: "zstd"
      acknowledgments: "all"
      max_events: 100
      disable_idempotent_write: false
      tls:
        status: true

With Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_redpanda
    type: redpanda
    properties:
      address: "redpanda.example.com"
      topic: "normalized-logs"
      field_format: "cim"
      compression: "snappy"

Redpanda Cloud

Configuration for Redpanda Cloud deployment:

targets:
  - name: redpanda_cloud
    type: redpanda
    properties:
      address: "seed-abc123.cloud.redpanda.com"
      port: 9093
      topic: "cloud-logs"
      algorithm: "scram-sha-256"
      username: "cloud-user"
      password: "cloud-password"
      compression: "zstd"
      tls:
        status: true
        insecure_skip_verify: false

Low Latency Configuration

Configuration optimized for minimal latency:

targets:
  - name: low_latency_redpanda
    type: redpanda
    properties:
      address: "redpanda.example.com"
      topic: "realtime-logs"
      compression: "none"
      acknowledgments: "leader"
      max_events: 100
      disable_idempotent_write: false

With Client ID

Configuration with client ID for tracking:

targets:
  - name: tracked_redpanda
    type: redpanda
    properties:
      address: "redpanda.example.com"
      topic: "tracked-logs"
      client_id: "datastream-producer-01"
      compression: "lz4"

Auto Topic Creation

Configuration with automatic topic creation enabled:

targets:
  - name: auto_topic_redpanda
    type: redpanda
    properties:
      address: "redpanda.example.com"
      topic: "dynamic-logs"
      allow_auto_topic_creation: true
      compression: "snappy"
      max_events: 500

Scheduled Batching

Configuration with scheduled batch delivery:

targets:
  - name: scheduled_redpanda
    type: redpanda
    properties:
      address: "redpanda.example.com"
      topic: "scheduled-logs"
      max_events: 5000
      interval: "5m"
      compression: "gzip"

Edge Deployment

Configuration for edge or self-hosted deployment:

targets:
  - name: edge_redpanda
    type: redpanda
    properties:
      address: "edge-redpanda.local"
      port: 9092
      topic: "edge-logs"
      compression: "lz4"
      max_events: 500
      max_bytes: 524288
      acknowledgments: "leader"