Persistent Storage

Synopsis

Persistent storage determines whether pipeline data survives a restart, and where the Director buffers that data. It is administered through two surfaces:

• The Director creation wizard and detail view in the web interface — covered in the sections below.
• The YAML configuration reference — for self-managed and infrastructure-as-code deployments. The persistent_storage block, its durability mode matrix, and the payload and pipeline bus settings are documented under Service Configuration.

The web interface exposes a subset of the YAML surface: it sets the processing mode and the external storage connection. Payload backends, stream size ceilings, and retention are YAML-only.

Availability

The persistent storage settings appear only when both of the following hold:

• The Director mode is Orchestrated.
• The processing mode is Persistent or Crash-resistant.

Managed and Self-managed Directors do not surface these settings. Selecting In-memory hides them, since no external storage is involved.

Accessing

Navigate to Home > Fleet Management > Directors and click Create new director. The processing mode and persistent storage settings are on the Director Setup step, below the Director name and mode selection.

To change the connection settings on an existing Director, open it from the Directors table, stay on the Director Details tab, and click Manage director details.

Processing Modes

Under Choose processing mode, three tiles determine how the Director processes and retains data:

Mode	Description
In-memory	Highest throughput with the lowest latency, using RAM as the buffer. Capacity is limited to available memory.
Persistent	Extends buffer capacity to your disk size and ensures data durability across process restarts. Built for handling longer outages without risking data loss.
Crash-resistant	Guarantees data recovery even after OS crashes or power loss. Built for pipelines where losing a single event is not an option.

The processing mode maps onto the persistent_storage.status and persistent_storage.crash_resistance flags described under Service Configuration. The web interface does not expose those flags directly.

Persistent and Crash-resistant are tier-gated. Without the corresponding entitlement, the tile is disabled and carries an Available in standard or Available in enterprise badge, and a See plans link appears next to the section description.

note

The default selection is Persistent when that tier is available, and In-memory otherwise. It is never Crash-resistant by default, even when that tier is available.

Storage Type

When the processing mode is Persistent or Crash-resistant, Choose persistent storage type offers two backends:

Type	Description
Apache Kafka	Cloud-agnostic distributed streaming. High availability, durability, and strong delivery guarantees.
Azure Event Hubs	Azure-native event streaming. Managed identity support and tight Azure integration.

Selecting a type reveals its connection form.

Apache Kafka

The following fields are used to configure the Kafka connection:

Field	Required	Default	Description
Broker Address (Optional)	N	0.0.0.0	Kafka broker address
Port	Y	9092	Kafka broker port. Must be between 1 and 65535
Topic Prefix	Y	-	Prefix used for the topics this Director writes to
Algorithm (Optional)	N	None	SASL mechanism. Leave as None to disable authentication. Accepts Plain, SCRAM-SHA-256, SCRAM-SHA-512, and GSSAPI (Kerberos)
Username (Optional)	N*	-	SASL username
Password (Optional)	N*	-	SASL password
TLS Encryption	N	Disabled	Enable TLS for the broker connection

* = Shown only when an Algorithm other than None is selected

Username and Password accept a value entered directly, an environment variable, or a Vault secret, selected from the dropdown beside each field.

Azure Event Hubs

The following fields are used to configure the Event Hubs connection:

Field	Required	Default	Description
Address	N	-	Event Hubs Kafka endpoint address
Topic Prefix	Y	-	Prefix used for the event hubs this Director writes to
Connection string	Y	-	Event Hubs connection string

Connection string accepts a value entered directly, an environment variable, or a Vault secret.

note

The Event Hubs form asks for three values where Kafka asks for seven. The Director supplies the rest when it saves the configuration: the SASL mechanism is set to plain, the username to $ConnectionString, and TLS is enabled. These are fixed for the Event Hubs Kafka endpoint and cannot be changed from the web interface.

Editing After Creation

Open the Director from the Directors table and click Manage director details on the Director Details tab.

The connection fields for the selected backend remain editable. The following are fixed at creation and display as read-only rows:

• Director mode
• Processing mode
• Persistent storage type

Changing any of these requires creating a new Director.