Azure Data Explorer
Synopsis
Reads tables from an Azure Data Explorer database over a bounded time window, through the cluster's Kusto REST query API. Each selected table is queried once per chunk of the window and the rows are paged into the pipeline.
This device is replay only: there is no streaming Azure Data Explorer collector, so the device type azdxreplay is the type itself rather than a replay of a sibling. Everything in Replay Devices — the time window, chunking, the run schedule, and the run controls — applies here.
Schema
- id: <numeric>
name: <string>
description: <string>
type: azdxreplay
tags: <string[]>
pipelines: <pipeline[]>
status: <boolean>
properties:
endpoint: <string>
database: <string>
tenant_id: <string>
client_id: <string>
client_secret: <string>
scope: <string>
stream: <string[]>
time_column: <string>
start_date: <string>
end_date: <string>
timezone: <string>
window_chunk: <numeric>
batch_size: <numeric>
interval: <numeric>
cron: <string>
Configuration
The following fields are used to define the device.
Device
| Field | Required | Default | Description |
|---|---|---|---|
id | Y | - | Unique numeric identifier |
name | Y | - | Device name |
description | N | - | Optional description of the device's purpose |
type | Y | - | Device type identifier (must be azdxreplay) |
tags | N | - | Array of labels for categorization |
pipelines | N | - | Array of preprocessing pipeline references |
status | N | true | Boolean flag to enable/disable the device |
Connection
| Field | Required | Default | Description |
|---|---|---|---|
endpoint | Y | - | Cluster URI. Must begin with https:// |
database | Y | - | Database within the cluster |
scope | N | <endpoint>/.default | Token scope requested for the cluster |
Authentication
| Field | Required | Default | Description |
|---|---|---|---|
tenant_id | N | - | Directory tenant of the service principal |
client_id | N | - | Application ID of the service principal |
client_secret | N | - | Client secret of the service principal |
Leave all three empty to authenticate with the managed or workload identity available to the Director. Credential fields support ${ENV_VAR} and $secret{...} token resolution.
Collection
| Field | Required | Default | Description |
|---|---|---|---|
stream | Y | - | Tables to replay. At least one is required |
time_column | N | TimeGenerated | Datetime column the window is applied to |
batch_size | N | 10000 | Rows per page within a chunk |
Window and Schedule
| Field | Required | Default | Description |
|---|---|---|---|
start_date | Y | - | Lower bound of the replay window |
end_date | N | - | Upper bound, exclusive. Unset means an open window that follows the present |
timezone | N | UTC | Zone used to read an offset-less bound |
window_chunk | N | 3600 | Seconds of the window covered by one query. Minimum 60 |
interval | N | 60 | Seconds between runs while the window still has work |
cron | N | - | Cron expression for the run, taking precedence over interval |
These behave exactly as described in Replay Devices, including the relative expressions accepted by start_date and end_date.
Details
Query Shape
Each table is queried once per chunk, bounded on time_column:
<table> | where <time_column> >= datetime(<earliest>) and <time_column> < datetime(<latest>)
The upper bound is exclusive, so adjacent chunks never overlap and no row is read twice. A table whose rows carry their timestamp in a column other than TimeGenerated needs time_column set accordingly; the whole device uses one column, so tables with different timestamp columns need separate devices.
Paging
Rows are read in pages of batch_size within each chunk. A single response is capped at 256 MB; a chunk whose result exceeds it needs a smaller window_chunk, a smaller batch_size, or both.
Credentials and Configuration Changes
Changing the endpoint, database, scope, credentials, time column, or table list is a breaking configuration change and restarts collection for the device. The window and the schedule are re-read live, so adjusting start_date, end_date, window_chunk, interval or cron does not restart it.
Examples
Basic
Replaying one month of a table with a service principal... | |
Managed Identity
Omitting the credentials uses the Director's managed identity... | |
Large Tables
Narrowing the chunk and the page size for a dense table, and running it nightly instead of continuously... | |