Lifecycle and Responsibility

Telemetry crosses two lines on its way from a source system to a destination. It enters DataStream when a device collects it, and it leaves when a destination acknowledges receipt. Between those two points every stage runs inside your own environment under DataStream control. Outside them, what a source emits and how a destination retains, indexes, and serves the data are properties of those systems. This page states where each line falls.

Provider → Device → Preprocessing → Pipeline → Postprocessing → Target → Consumer

Stage Ownership

Stage	What happens	Controlled by	Notes
Generation	Source systems emit logs, metrics, traces, and events	Source system	DataStream does not control what a source emits, or when
Collection	Push-type devices listen for incoming data and pull-type devices poll the source, across protocols, cloud platforms, message queues, databases, and agents; OTLP carries logs, metrics, and traces	DataStream	Runs in your infrastructure. Agents run the same pipeline engine as the Director, so filtering and masking can happen at the point of collection
Preprocessing	Parsing, normalization to a canonical schema, schema validation, and per-source drift detection	DataStream	Drift findings are written to the record and can be routed to notification processors
Pipeline processing	Transformation, enrichment, redaction and masking against curated rulesets, and volume reduction through deduplication, suppression, sampling, aggregation, and cardinality limits	DataStream	All processing is local to your environment
Buffering	Per-target worker pools, retries with exponential backoff, and persistent or crash-resistant storage	DataStream	The disk queue and the durable streams each hold 48 hours by default; both are configurable
Delivery	Criteria-based routing, per-destination transformation, fan-out, and staged rollout through a pipeline's staging version	DataStream, to acknowledged delivery	Responsibility ends here: a record is DataStream's until the destination acknowledges it
Retention, indexing, and query	Storage duration, index strategy, and search at the SIEM, data lake, or object store	Destination system	DataStream chooses the destination by routing policy but manages nothing after delivery
Rotation and expiry at source	Log rotation and retention on the originating system	Source system	Runs in parallel with the chain rather than after it

Where Control Begins and Ends

Control begins at collection. A device running on your infrastructure is the first component DataStream owns; everything before it belongs to the source system.

Control ends at acknowledged delivery. Until a destination confirms receipt, the record is DataStream's responsibility; from that point, retention, indexing, and search belong to the destination. Between the two lines — parsing, normalization, enrichment, redaction, reduction, buffering, routing — every stage runs inside your own environment. The deployment and network model this rests on is described in Architecture.

When a destination does not acknowledge, the record stays in the durable queue and is redelivered to the same destination until it succeeds or its retention window expires.

Observing the Boundary

A responsibility boundary is only useful if you can see where a record stands relative to it. Three surfaces show that:

• Delivery counters per destination, reported alongside the volume each pipeline takes in and sends out, on the Stats Overview page.
• Queue depth, backpressure, and consumer lag, available as alert rules that raise an alert when a destination stops keeping up.
• The durable backlog itself, which holds 48 hours by default. queue.limit governs the on-disk queue and persistent_storage.max_age the durable streams; both are documented in the service configuration reference.

The Control Plane and Log Content

The control plane carries configuration, fleet state, and telemetry counters. It performs no data processing, and in normal operation it does not receive log content: the Director holds a single outbound connection to it, and nothing connects inward.

One path is an exception, and it is operator-initiated. Live Data captures real records at a chosen processing stage so they can be read in the web interface; Console Logs does the same for the Director's internal debug messages. Only the first carries log content. Both are bounded sessions rather than continuous tails: a capture runs for at most 60 seconds, collects at most 5,000 events, and stops on its own when either limit is reached. Nothing is captured unless an operator starts one.

Outside the Boundary

Two parts of the lifecycle sit outside the pipeline by design.

Before collection, the source system decides what to emit, at what verbosity, and how long to keep its own copy. DataStream can filter and reduce what it has collected, but it cannot make a source emit a field the source never wrote.

After delivery, the destination decides storage duration, index strategy, and query behavior. DataStream influences this at one remove: routing policy chooses which destination a record reaches, so compliance data can be directed to long-term object storage while operational data goes to the SIEM. That is a choice of destination, not management of the data once it arrives.