Azure Blob Storage

Microsoft Azure Long Term Storage

Synopsis

Creates a target that writes log messages to Azure Blob Storage with support for various file formats, authentication methods, and retry mechanisms. Inherits file format capabilities from the base target.

Schema

- name: <string>
  description: <string>
  type: azblob
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    account: <string>
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    container: <string>
    name: <string>
    type: <string>
    extension: <string>
    compression: <string>
    schema: <string>
    field_format: <string>
    no_buffer: <boolean>
    max_retry: <numeric>
    retry_interval: <numeric>
    timeout: <numeric>
    max_size: <numeric>
    batch_size: <numeric>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be azblob
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Azure

Field	Required	Default	Description
account	Y		Azure storage account name
tenant_id	N*	-	Azure tenant ID (required unless using managed identity)
client_id	N*	-	Azure client ID (required unless using managed identity)
client_secret	N*	-	Azure client secret (required unless using managed identity)
container	N	"vmetric"	Blob container name

* = Conditionally required (see notes in the Azure section above)

Connection

Field	Required	Default	Description
max_retry	N	5	Maximum number of upload retries
retry_interval	N	10	Base interval between retries in seconds
timeout	N	30	Connection timeout in seconds
max_size	N	0	Maximum file size in bytes
batch_size	N	100000	Maximum number of messages per file

note

When max_size is reached, the current file is uploaded to blob storage and a new file is created. For unlimited file size, set the field to 0.

Files

The following fields can be used for files:

Field	Required	Default	Description
name	N	"vmetric.{{.Timestamp}}.{{.Extension}}"	Blob name template
type	N	"json"	File format (json, multijson, avro, parquet)
extension	N	Matches type	Custom file extension
compression	N	"zstd"	Compression algorithm
schema	N	-	Data schema for Avro/Parquet formats or schema template name
no_buffer	N	false	Disable write buffering
field_format	N	-	Data normalization format. See applicable Normalization section

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to Azure (testing)

Details

The Azure Blob Storage target supports writing to different containers with various file formats and schemas. When using the SystemS3 field in your logs, the value will be used to route the message to the appropriate container.

The target supports the following built-in schema templates for structured data formats:

• {{Syslog}} - Standard schema for Syslog messages
• {{CommonSecurityLog}} - Schema compatible with Common Event Format (CEF)

Templates

The following template variables can be used in the blob name:

Variable	Description	Example
{{.Year}}	Current year	2024
{{.Month}}	Current month	01
{{.Day}}	Current day	15
{{.Timestamp}}	Current timestamp in nanoseconds	1703688533123456789
{{.Format}}	File format	json
{{.Extension}}	File extension	json
{{.Compression}}	Compression type	zstd
{{.TargetName}}	Target name	my_logs
{{.TargetType}}	Target type	azblob
{{.Table}}	Container name	logs

Formats

Format	Description
json	Each log entry is written as a separate JSON line (JSONL format)
multijson	All log entries are written as a single JSON array
avro	Apache Avro format with schema
parquet	Apache Parquet columnar format with schema

Compression

Format	Compression Options
JSON/MultiJSON	zstd (default)
Avro	null, deflate, snappy, zstd
Parquet	uncompressed, gzip, snappy, zstd, brotli, lz4

warning

Failed uploads are retried automatically based on the max_retry and retry_interval settings. The retry_interval setting is based on an exponential backoff which is retry_interval * 2 ^ attempt, therefore each retry will wait longer than the previous one.

note

Files with no messages (i.e. with counter=0) are automatically skipped during upload.

Examples

The following upload configurations are available.

JSON

The minimum configuration for a JSON blob storage:

targets:
  - name: basic_blob
    type: azblob
    properties:
      account: "mystorageaccount"
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"

Parquet

Configuration for daily partitioned Parquet files:

targets:
  - name: parquet_blob
    type: azblob
    properties:
      account: "mystorageaccount"
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      container: "logs"
      type: "parquet"
      compression: "zstd"
      name: "logs/year={{.Year}}/month={{.Month}}/day={{.Day}}/data_{{.Timestamp}}.parquet"
      max_size: 536870912  # 512MB

High Reliability

Configuration with enhanced retry settings:

targets:
  - name: reliable_blob
    type: azblob
    pipelines:
      - checkpoint
    properties:
      account: "mystorageaccount"
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      max_retry: 10
      retry_interval: 30
      timeout: 60
      no_buffer: true

Debug Configuration

Configuration with debugging enabled:

targets:
  - name: debug_blob
    type: azblob
    properties:
      account: "mystorageaccount"
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      debug:
        status: true
        dont_send_logs: true  # Test mode that doesn't actually upload