Azure Blob Storage
Synopsis
Azure Blob Storage device reads files from an Azure Blob Storage container. By default it dequeues messages from an Azure Storage Queue that notifies the device of new blobs. In poll mode it lists the container on a schedule instead, with no queue.
Schema
- id: <numeric>
name: <string>
description: <string>
type: azblob
tags: <string[]>
pipelines: <pipeline[]>
status: <boolean>
properties:
container_name: <string>
connection_string: <string>
tenant_id: <string>
client_id: <string>
client_secret: <string>
account: <string>
blob_size: <numeric>
mode: <enum>
prefix: <string>
path_prefix: <string>
file_name_filter: <string>
timezone: <string>
workers: <numeric>
max_size_archive_bytes: <numeric>
max_files_in_archive: <numeric>
timeout: <numeric>
batch_size: <numeric>
pollers: <numeric>
visibility_timeout: <numeric>
interval: <numeric>
cron: <string>
start_date: <string>
end_date: <string>
max_retries: <numeric>
Configuration
The following fields are used to define the device:
Device
| Field | Required | Default | Description |
|---|---|---|---|
id | Y | Unique numeric identifier | |
name | Y | Device name | |
description | N | - | Optional description |
type | Y | Device type identifier (must be azblob) | |
tags | N | - | Array of labels for categorization |
pipelines | N | - | Array of preprocessing pipeline references |
status | N | true | Enable/disable the device |
Connection
The device requires either a connection string or a service principal with account name. container_name is required in all cases.
| Field | Required | Default | Description |
|---|---|---|---|
container_name | Y | Container to read. In queue mode it is also the name of the Azure Storage Queue that notifies the device of new blobs | |
connection_string | Y* | Azure Storage connection string for authentication | |
tenant_id | Y* | Azure tenant ID for service principal authentication | |
client_id | Y* | Azure client ID for service principal authentication | |
client_secret | Y* | Azure client secret for service principal authentication | |
account | Y* | Azure storage account name for service principal authentication | |
blob_size | N | 100000000 | Maximum blob size in bytes to process (100 MB) |
mode | N | queue | Ingestion mode: queue or poll. See Ingestion Modes |
prefix | N | - | Restrict ingestion to keys under this prefix. In poll mode it may carry date tokens |
path_prefix | N | - | Read in place of prefix when prefix is not set, which is how a device saved by an older console applies its prefix. A value the mode cannot use, date tokens or a stray % in queue mode, or a malformed template in poll mode, is ignored: the whole container is read and the Director log carries a warning, path_prefix "<value>" is ignored, so the whole container is read: <reason>; set prefix instead. The same value in prefix is rejected instead |
file_name_filter | N | .* | Regular expression a blob name must match to be downloaded. An invalid expression is ignored and the previous filter is kept |
timezone | N | UTC | IANA name or offset stamped on every record. In poll mode it also applies to an offset-less or relative start_date and end_date |
workers | N | max(NumCPU/4, 1) | Concurrent download and decode workers, 1-64 |
max_size_archive_bytes | N | 104857600 | Largest total size of the files extracted from one archive, in bytes; larger archives are skipped. 0 or an empty value uses the default |
max_files_in_archive | N | 100 | Most files extracted from one archive; archives with more files are skipped. 0 or an empty value uses the default |
timeout | N | 5 | queue mode only. Idle backoff in seconds when the notification queue is empty (1-3600) |
batch_size | N | 32 | queue mode only. Maximum queue messages retrieved per dequeue (1-32; Azure caps at 32) |
pollers | N | 2 | queue mode only. Concurrent loops that dequeue notifications (1-16) |
visibility_timeout | N | 300 | queue mode only. Seconds a dequeued notification stays hidden while its blob is read and delivered (1-604800). Set it longer than that takes, or the blob is read twice |
interval | N | 60 | poll mode only. Seconds between list cycles. See Poll Mode Properties |
cron | N | - | poll mode only. Cron schedule for the list cycle. See Poll Mode Properties |
start_date | N | - | poll mode only. Lower bound of the listing window. See Poll Mode Properties |
end_date | N | - | poll mode only. Upper bound of the listing window. See Poll Mode Properties |
max_retries | N | 0 | poll mode only. Consecutive cycles a blob may fail before it is skipped. 0 means never give up |
A numeric value outside its range is ignored and the default applies.
* = Conditionally required: either connection_string or the combination of tenant_id, client_id, client_secret, and account must be provided.
Ingest Buffer
| Field | Required | Default | Description |
|---|---|---|---|
max_buffer_size | N | 128MB | Ingest buffer admission cap. Also sizes this listener's ingest buffers: 3/8 of this value at start, growing to at most 3x |
max_buffer_size is the ingest-buffer admission cap, and is distinct from any buffer_size field above, which sizes the network read buffer. Left unset it is 128MB, whose buffers take 48 MiB of heap per listener worker at start and grow to at most 384 MiB as bursts need the room — and with reuse: true a device runs one worker per listener. Size strings are binary, and 32MB and 32MiB are exact synonyms.
See Listener Memory Sizing for the arithmetic, the sizing table, and what happens when the cap is exhausted.
Details
Ingestion Modes
This device offers two ingestion modes, selected with mode:
mode | Behaviour |
|---|---|
queue | Default. The device consumes notifications from the bucket's event queue and fetches each object as it is announced. |
poll | The device lists the bucket on a schedule and ingests what it has not seen before. No queue is needed. |
Any other value fails with mode field not valid.
Poll mode suits a bucket with no notification wiring, and a historical backfill, since a listing can be bounded to a date range. It costs a list operation per cycle, so it is the wrong choice for a high-rate bucket that already has a queue.
Poll Mode Properties
These are read only when mode: poll is set.
| Field | Required | Default | Description |
|---|---|---|---|
interval | N | 60 | Seconds between list cycles. When cron is set as well, a cycle starts when either is due. The default applies only when cron is unset |
cron | N | - | Cron expression for the list cycle. Alone, it is the only schedule; with interval, a cycle starts when either is due. An invalid expression is logged and the interval schedule is used instead |
workers | N | max(NumCPU/4, 1) | Concurrent download and decode workers, 1–64 |
max_retries | N | 0 | Consecutive cycles an object may fail transiently before the poller gives up on it and moves the cursor past it. 0 means never give up |
start_date | N | - | Lower bound of the listing window. Accepts Unix seconds, RFC 3339, a timestamp without an offset (read in timezone), or a relative expression. Unset or negative means the whole bucket |
end_date | N | - | Upper bound, exclusive. Unset means unbounded; a negative value is an error, and a value at or before start_date is rejected |
timezone | N | UTC | Zone used to interpret an offset-less start_date or end_date |
The Cursor
Poll mode keeps a since-cursor so a restart or a cluster failover does not re-ingest what it already has. The cursor is scoped to the listing target — the bucket and prefix together — so changing either starts a fresh one.
A stored cursor takes precedence over start_date, but only while the window is unchanged: editing start_date, end_date or prefix discards the cursor and re-evaluates the new window. A bounded window that has been fully drained stops listing and idles until the window changes.
Date Tokens in prefix
In poll mode prefix may carry date tokens, so a listing walks only the partitions that fall inside its window: %Y, %y, %m, %d, %j, %H, and %% for a literal percent. The finest token present sets the partition step.
A token-bearing prefix needs a bounded start_date — there is no way to walk partitions from the beginning of a bucket — and a window spanning too many partitions is rejected rather than expanded.
Date tokens are rejected in queue mode. A queue subscription filters on a fixed key prefix, which cannot be re-expanded per cycle, so a prefix containing %Y and the like fails at apply time. Write %% where a literal percent is wanted.
IAM Permissions
When using service principal authentication, the following Azure RBAC roles are required:
| Azure Role | Scope | Purpose |
|---|---|---|
Storage Blob Data Reader | Storage Account or Container | Read blobs and blob properties |
Storage Queue Data Message Processor | Storage Account or Queue | Dequeue and delete queue messages |
The device validates connectivity at startup by reading blob service properties and queue metadata.
Queue-Based Notification
The device uses an Azure Storage Queue as a notification mechanism. The queue name is specified by container_name. When a blob lands in the storage account, Azure emits a queue message pointing to the blob. The device dequeues these messages and downloads the referenced blobs for processing.
Authentication
When connection_string is provided, the device builds both the blob and queue clients directly from the connection string. When account is provided instead, the device constructs clients using the service principal credentials and builds URLs of the form https://<account>.blob.core.windows.net/ and https://<account>.queue.core.windows.net/<container_name>/.
Replay
The azblobreplay device type is this collector permanently in poll mode. It lists the container for a bounded time window and ingests the blobs it finds, without an Event Grid storage queue.
Every property behaves as documented under Poll Mode above, including start_date, end_date, timezone, the date tokens in prefix, and the cursor. The only configuration difference is type: mode may be omitted, since a replay can only poll, and setting it to queue is rejected.
What the replay type adds is a reported run — live progress, a run history, and the pause, resume, cancel and run-now controls. See Replay Devices.
Examples
Connection String Authentication
Connecting with an Azure Storage connection string... | |
Service Principal Authentication
Connecting with service principal credentials... | |
Large Blob Processing
Increasing the blob size limit for large files... | |