Replay Devices

Synopsis

A replay device re-ingests a bounded window of history from a source that DataStream already collects from. It runs the same collector as its streaming sibling — the same module, the same credentials, the same decoding — with one difference: instead of following new data as it arrives, it works through a time window you set, from start_date to end_date, and stops when it reaches the end.

Each replay is a device type of its own rather than a mode flag on the streaming type. A replay device therefore has its own entry in the interface, its own instance list, its own counts, and its own run history, and it can be configured and run without disturbing the streaming device it mirrors.

Replays serve onboarding a new target with historical data, re-processing a period through a corrected pipeline, and backfilling a gap left by an outage.

Device Types

Replay type	Streaming sibling	Source of the window
awss3replay	awss3	Object LastModified
azblobreplay	azblob	Blob LastModified
gcsreplay	gcs	Object LastModified
mssqlreplay	mssql	The column each query bounds
mysqlreplay	mysql	The column each query bounds
postgresreplay	postgres	The column each query bounds
oraclereplay	oracle	The column each query bounds
ccfreplay	ccf	The connector's query window
azlogsreplay	azlogs	TimeGenerated
azdxreplay	—	The configured time_column

azdxreplay has no streaming sibling: Azure Data Explorer is collected only as a replay.

Two Families

Replay devices divide by how they reach the window, and the division decides which properties apply.

Listing replays — awss3replay, azblobreplay, gcsreplay — list the bucket or container for the window and ingest the objects whose modification time falls inside it. They are their streaming collector permanently in poll mode, so mode may be omitted and queue is rejected. Every other property behaves as documented under Poll Mode on the sibling's page.

Query replays — the database, CCF, Azure Monitor Logs and Azure Data Explorer replays — cut the window into chunks and run each query, connector, or table once per chunk. These are the types that use window_chunk.

The Time Window

Field	Required	Default	Description
start_date	Y for query replays	-	Lower bound. A listing replay without one lists the whole bucket; a query has no whole-bucket extent, so a query replay requires it
end_date	N	-	Upper bound, exclusive. Unset means an open window that keeps catching up to the present on every run. A negative value is an error, and a value at or before start_date is rejected
timezone	N	UTC	Zone used to read an offset-less bound, and to evaluate calendar arithmetic in a relative expression

Both bounds accept Unix seconds, an RFC 3339 timestamp with an offset, a timestamp without an offset (read in timezone), or a relative expression.

Relative Expressions

A relative expression is evaluated from the present moment, left to right:

[now] ( [+|-]<integer><unit> | @<snap> )*

Units are s, m, h, d, w, mon, q and y, and each accepts its longer spellings such as min, hour, month, quarter and year. A snap written @<unit> rounds down to the start of that unit; @w, @w0 and @w7 snap to Sunday, while @w1 through @w6 snap to Monday through Saturday.

Expression	Resolves to
-1mon@mon	The start of last calendar month
@mon	The start of this calendar month
-7d@d	Midnight seven days ago
@d	Midnight today
-1q@q	The start of last quarter

Writing start_date: "-1mon@mon" with end_date: "@mon" therefore replays exactly the previous calendar month.

Day, week, month, quarter and year arithmetic is calendar arithmetic and is evaluated in timezone; seconds, minutes and hours are absolute durations. A bare integer is read as a Unix epoch rather than an expression, so an expression must carry a unit or a snap.

An expression is resolved once, when it is applied. A device configured with -1mon@mon does not roll forward to a new month on its own — editing the window is what re-arms it.

Chunking

A query replay does not ask its source for the whole window at once. It divides the window into chunks and runs each unit — a query, a connector, a table — once per chunk, bounding the request to that chunk's start and end.

Field	Required	Default	Description
window_chunk	N	3600	Seconds of the window covered by one execution of one unit. Minimum 60

ccfreplay is the exception: it defaults to each connector's own queryWindowInMin, the window size the connector was written against, and window_chunk overrides that.

A cursor is persisted per unit after each chunk it completes, so a chunk is never run twice. If a chunk fails, that unit stops for the rest of the run and resumes at the same chunk on the next one — the chunks behind it wait rather than being skipped.

One run executes at most 2000 chunks. A window that plans more is not rejected; the remainder carries into the next run.

Execution

A replay device runs on a schedule of its own, and that schedule is a property of the device rather than of anything inside it. On a query replay this matters: the per-input cron and interval of the collector's own definitions are ignored, because a run walks every unit's pending chunks in one pass instead of running each unit as an independent poller.

Field	Required	Default	Description
interval	N	60	Seconds between runs while the window still has work. The device starts on its first tick
cron	N	-	Cron expression for the run. Takes precedence over interval

With interval, the device begins immediately and repeats until the window is drained. With cron, it runs only on the schedule.

Once every unit has reached end_date the device is drained: it stops working and idles. It does not run again until the window changes, and changing start_date or end_date re-arms it from the new window start. An open-ended window is never drained, since there is always more present to catch up to.

Runs and Controls

Each firing of the schedule is a run, reported to the platform while it is in progress and listed afterwards on the device's Replay runs tab.

A run carries one of five statuses:

Status	Meaning
running	The run is working through its chunks or objects
paused	An operator paused it; it holds its position
completed	The run finished the work it planned
cancelled	An operator cancelled it before it finished
failed	The run ended on an error

It also records what started it: interval, cron, or manual for an operator-triggered run.

Four actions are available on a running replay — pause, resume, cancel, and run now. They reach the collector through the Director running it, so a device that is disabled, still starting, or placed on a different Director has no running poller to receive them.

Progress is reported as the share of planned work completed — chunks for a query replay, listed objects for a listing replay — together with counts of what succeeded, what was skipped, and what failed. A listing replay additionally reports the bytes it has listed and processed.

Discovery

Before a replay runs, a preview enumerates the units it will execute and the chunks it plans for each. Units that cannot take part are listed with the reason, so a replay that would quietly do nothing is visible in advance rather than after the fact. A unit is excluded when it cannot be bounded by time:

• A database query that does not bind {{earliest}} and {{latest}}, or one that tracks a {{cursor}} column, since a replay has no high-watermark to advance
• A CCF connector paging by PersistentToken or PersistentLinkHeader, which cannot be resumed at an arbitrary point in time

Isolation from the Streaming Device

A replay device never touches the state of the sibling it mirrors. The streaming device's checkpoints, cursors, tokens and link state are left exactly as they are, and the two can run at the same time against the same source.

A replay carries the pipeline and route configuration of the replay device itself, so a backfill can be sent somewhere other than the live stream by attaching different pipelines or routing it to a different target.