Replay Devices
Synopsis
A replay device re-ingests a bounded window of history from a source that DataStream already collects from. It runs the same collector as its streaming sibling — the same module, the same credentials, the same decoding — with one difference: instead of following new data as it arrives, it works through a time window you set, from start_date to end_date, and stops when it reaches the end.
Each replay is a device type of its own rather than a mode flag on the streaming type. A replay device therefore has its own entry in the interface, its own instance list, its own counts, and its own run history, and it can be configured and run without disturbing the streaming device it mirrors.
Replays serve onboarding a new target with historical data, re-processing a period through a corrected pipeline, and backfilling a gap left by an outage.
Device Types
| Replay type | Streaming sibling | Source of the window |
|---|---|---|
awss3replay | awss3 | Object LastModified |
azblobreplay | azblob | Blob LastModified |
gcsreplay | gcs | Object LastModified |
mssqlreplay | mssql | The column each query bounds |
mysqlreplay | mysql | The column each query bounds |
postgresreplay | postgres | The column each query bounds |
oraclereplay | oracle | The column each query bounds |
ccfreplay | ccf | The connector's query window |
azlogsreplay | azlogs | TimeGenerated |
azdxreplay | — | The configured time_column |
azdxreplay has no streaming sibling: Azure Data Explorer is collected only as a replay.
Two Families
Replay devices divide by how they reach the window, and the division decides which properties apply.
Listing replays — awss3replay, azblobreplay, gcsreplay — list the bucket or container for the window and ingest the objects whose modification time falls inside it. They are their streaming collector permanently in poll mode, so mode may be omitted and queue is rejected. Every other property behaves as documented under Poll Mode on the sibling's page.
Query replays — the database, CCF, Azure Monitor Logs and Azure Data Explorer replays — cut the window into chunks and run each query, connector, or table once per chunk. These are the types that use window_chunk.
The Time Window
| Field | Required | Default | Description |
|---|---|---|---|
start_date | Y for query replays | - | Lower bound. A listing replay without one lists the whole bucket; a query has no whole-bucket extent, so a query replay requires it |
end_date | N | - | Upper bound, exclusive. Unset means an open window that keeps catching up to the present on every run. A negative value is an error, and a value at or before start_date is rejected |
timezone | N | UTC | Zone used to read an offset-less bound, and to evaluate calendar arithmetic in a relative expression |
Both bounds accept Unix seconds, an RFC 3339 timestamp with an offset, a timestamp without an offset (read in timezone), or a relative expression.
Relative Expressions
A relative expression is evaluated from the present moment, left to right:
[now] ( [+|-]<integer><unit> | @<snap> )*
Units are s, m, h, d, w, mon, q and y, and each accepts its longer spellings such as min, hour, month, quarter and year. A snap written @<unit> rounds down to the start of that unit; @w, @w0 and @w7 snap to Sunday, while @w1 through @w6 snap to Monday through Saturday.
| Expression | Resolves to |
|---|---|
-1mon@mon | The start of last calendar month |
@mon | The start of this calendar month |
-7d@d | Midnight seven days ago |
@d | Midnight today |
-1q@q | The start of last quarter |
Writing start_date: "-1mon@mon" with end_date: "@mon" therefore replays exactly the previous calendar month.
Day, week, month, quarter and year arithmetic is calendar arithmetic and is evaluated in timezone; seconds, minutes and hours are absolute durations. A bare integer is read as a Unix epoch rather than an expression, so an expression must carry a unit or a snap.
An expression is resolved once, when it is applied. A device configured with -1mon@mon does not roll forward to a new month on its own — editing the window is what re-arms it.
Chunking
A query replay does not ask its source for the whole window at once. It divides the window into chunks and runs each unit — a query, a connector, a table — once per chunk, bounding the request to that chunk's start and end.
| Field | Required | Default | Description |
|---|---|---|---|
window_chunk | N | 3600 | Seconds of the window covered by one execution of one unit. Minimum 60 |
ccfreplay is the exception: it defaults to each connector's own queryWindowInMin, the window size the connector was written against, and window_chunk overrides that.
A cursor is persisted per unit after each chunk it completes, so a chunk is never run twice. If a chunk fails, that unit stops for the rest of the run and resumes at the same chunk on the next one — the chunks behind it wait rather than being skipped.
One run executes at most 2000 chunks. A window that plans more is not rejected; the remainder carries into the next run.
Execution
A replay device runs on a schedule of its own, and that schedule is a property of the device rather than of anything inside it. On a query replay this matters: the per-input cron and interval of the collector's own definitions are ignored, because a run walks every unit's pending chunks in one pass instead of running each unit as an independent poller.
| Field | Required | Default | Description |
|---|---|---|---|
interval | N | 60 | Seconds between runs while the window still has work. The device starts on its first tick |
cron | N | - | Cron expression for the run. Takes precedence over interval |
With interval, the device begins immediately and repeats until the window is drained. With cron, it runs only on the schedule.
Once every unit has reached end_date the device is drained: it stops working and idles. It does not run again until the window changes, and changing start_date or end_date re-arms it from the new window start. An open-ended window is never drained, since there is always more present to catch up to.
Runs and Controls
Each firing of the schedule is a run, reported to the platform while it is in progress and listed afterwards on the device's Replay runs tab.
A run carries one of five statuses:
| Status | Meaning |
|---|---|
running | The run is working through its chunks or objects |
paused | An operator paused it; it holds its position |
completed | The run finished the work it planned |
cancelled | An operator cancelled it before it finished |
failed | The run ended on an error |
On a query replay, a run whose every attempted chunk failed is failed, and it records the first failure as its error. A run in which some chunks failed and others succeeded is still completed:
| A query replay run where… | Status | Recorded error |
|---|---|---|
| Every chunk it tried failed | failed | could not read <unit> [<from>, <to>): <reason>, the first failure, summarized as under Connection State |
| Some chunks failed and others succeeded | completed | None. The failed and skipped counts show what is behind, and those units resume on the next run |
| An operator cancelled it | cancelled | - |
A failed run does not carry straight into the next one, even when it stopped at the 2000-chunk cap: the next run waits for the schedule.
It also records what started it: interval, cron, or manual for an operator-triggered run.
Four actions are available on a running replay — pause, resume, cancel, and run now. They reach the collector through the Director running it, so a device that is disabled, still starting, or placed on a different Director has no running poller to receive them.
Progress is reported as the share of planned work completed — chunks for a query replay, listed objects for a listing replay — together with counts of what succeeded, what was skipped, and what failed. A listing replay additionally reports the bytes it has listed and processed.
Discovery
Before a replay runs, a preview enumerates the units it will execute and the chunks it plans for each. Units that cannot take part are listed with the reason, so a replay that would quietly do nothing is visible in advance rather than after the fact. A unit is excluded when it cannot be bounded by time:
- A database query that does not bind
{{earliest}}and{{latest}}, or one that tracks a{{cursor}}column, since a replay has no high-watermark to advance - A CCF connector paging by
PersistentTokenorPersistentLinkHeader, which cannot be resumed at an arbitrary point in time
Connection State
A query replay's connection state follows what its runs find, so a replay that cannot read anything does not look healthy. This applies to the query replays only — mssqlreplay, mysqlreplay, postgresreplay, oraclereplay, ccfreplay, azlogsreplay and azdxreplay. The listing replays are unchanged.
| When | State | Message |
|---|---|---|
| The device starts, before its first run | Connecting | <module> replay of <device> starting; the first run reports whether the source answers |
A run tried chunks and none succeeded, because the source refused the credentials (a refused token, a 401 or a 403) | Authentication error | <module> replay of <device> could not read <unit> [<from>, <to>): <reason>; retried on the next run |
A run tried chunks and none succeeded, for any other reason (the source is unreachable, a query error, a 500) | Connection error | The same message as the authentication error |
| A run in which at least one chunk succeeded | Connected | The run's progress, for example Replaying <start> to <end>: cursor at <cursor> (<n>% of the window). When some units failed, it adds Could not read <units>; retried on the next run. |
The window does not resolve, for example because start_date is missing | Configuration error | <module> replay of <device> is not running: the replay window is not valid: <reason> |
- One failing unit does not flip the device. A unit is one query, connector or table. When one fails and another succeeds, the device stays Connected and its message names the unit that is behind. Only a run in which nothing succeeded reports an error.
- A failure stands until a chunk succeeds. A run that tries nothing, such as one cancelled before its first chunk, leaves the error in place. A run with nothing pending, such as one over a drained window after a restart, reports Connected with its progress.
- The first state is set by the first run. On an
intervalschedule that is within about a second of the device starting; on acronschedule it is at the first cron fire. Until then the device reads Connecting. - The
<reason>is a short summary: the error's first line, at most 300 characters. It never includes the URL of a failed request or the body of an API response, and credential-shaped values such aspassword=,token:orBearer …are masked. The full error, body included, is in the Director log for that chunk. - A failed chunk is retried, not skipped. The unit resumes at the same chunk on the next run, as described under Chunking.
- The state is written only when it changes, so the same failure repeating on every run does not churn the device's history.
Isolation from the Streaming Device
A replay device never touches the state of the sibling it mirrors. The streaming device's checkpoints, cursors, tokens and link state are left exactly as they are, and the two can run at the same time against the same source.
A replay carries the pipeline and route configuration of the replay device itself, so a backfill can be sent somewhere other than the live stream by attaching different pipelines or routing it to a different target.