Persistent Storage

Synopsis

Persistent storage determines whether pipeline data survives a restart, and where the Director buffers that data. It is administered through two surfaces:

• The Director creation wizard and detail view in the web interface — covered in the sections below.
• The YAML configuration reference — for self-managed and infrastructure-as-code deployments. The persistent_storage block, its durability mode matrix, and the payload and pipeline bus settings are documented under Service Configuration.

The web interface exposes a subset of the YAML surface: it sets the processing mode, the external storage connection, and three buffer settings -- retention, the queue size limit and the inline payload size (see Buffer Settings). The payload backend type, the per-queue size keys under persistent_storage.streams and the stats stream settings are YAML-only.

Availability

The storage type and connection settings appear only when both of the following hold:

• The Director mode is Orchestrated or Serverless.
• The processing mode is Persistent or Crash-resistant.

Managed and Self-managed Directors keep their persistent queue on the host's own disk, so they have no external storage to connect to. Selecting In-memory hides the settings, since no external storage is involved.

An Orchestrated Director always uses the configured broker as its pipeline bus. A Serverless Director does so only when its connection names a broker address; with the address left empty it keeps its queue inside the Director.

The Buffer settings block is broader than the connection form: it appears in every Director mode once the processing mode is Persistent or Crash-resistant, because every mode then keeps a persistent queue. See Buffer Settings.

Accessing

Navigate to Home > Fleet Management > Directors and click Create new director. The processing mode and persistent storage settings are on the General settings step, below the Director name and mode selection.

To change the connection or buffer settings on an existing Director, open it from the Directors table, stay on the Director overview tab, and click Manage general settings.

Processing Modes

Under Choose processing mode, three tiles determine how the Director processes and retains data:

Mode	Description
In-memory	Highest throughput with the lowest latency, using RAM as the buffer. Capacity is limited to available memory.
Persistent	Extends buffer capacity to your disk size and ensures data durability across process restarts. Built for handling longer outages without risking data loss.
Crash-resistant	Guarantees data recovery even after OS crashes or power loss. Built for pipelines where losing a single event is not an option.

The processing mode maps onto the persistent_storage.status and persistent_storage.crash_resistance flags described under Service Configuration. The web interface does not expose those flags directly.

Persistent and Crash-resistant are tier-gated. Without the corresponding entitlement, the tile is disabled and carries an Available in standard or Available in enterprise badge, and a See plans link appears next to the section description.

note

The default selection is Persistent when that tier is available, and In-memory otherwise. It is never Crash-resistant by default, even when that tier is available.

The processing mode is fixed for the life of the Director process. A Director that receives its configuration from the platform applies the mode from the first configuration delivered to it, which is why a brand-new one waits for that delivery before it builds its pipeline rather than starting on a guess — see Director Waiting for Its First Configuration.

Storage Type

When the processing mode is Persistent or Crash-resistant, Choose persistent storage type offers two backends:

Type	Description
Apache Kafka	Cloud-agnostic distributed streaming. High availability, durability, and strong delivery guarantees.
Azure Event Hubs	Azure-native event streaming. Managed identity support and tight Azure integration.

Selecting a type reveals its connection form.

Apache Kafka

The following fields are used to configure the Kafka connection:

Field	Required	Default	Description
Broker Address (Optional)	N	0.0.0.0	Kafka broker address
Port	Y	9092	Kafka broker port. Must be between 1 and 65535
Topic prefix	Y	-	Prefix used for the topics this Director writes to
Algorithm (Optional)	N	None	SASL mechanism. Leave as None to disable authentication. Accepts Plain, SCRAM-SHA-256, SCRAM-SHA-512, and GSSAPI (Kerberos)
Username (Optional)	N*	-	SASL username
Password (Optional)	N*	-	SASL password
TLS encryption	N	Disabled	Enable TLS for the broker connection

* = Shown only when an Algorithm other than None is selected

Username and Password accept a value entered directly, an environment variable, or a Vault secret, selected from the dropdown beside each field.

Azure Event Hubs

The following fields are used to configure the Event Hubs connection:

Field	Required	Default	Description
Address	N	-	Event Hubs Kafka endpoint address
Topic prefix	Y	-	Prefix used for the event hubs this Director writes to
Connection string	Y	-	Event Hubs connection string

Connection string accepts a value entered directly, an environment variable, or a Vault secret.

note

The Event Hubs form asks for three values where Kafka asks for seven. The Director supplies the rest when it saves the configuration: the SASL mechanism is set to plain, the username to $ConnectionString, and TLS is enabled. These are fixed for the Event Hubs Kafka endpoint and cannot be changed from the web interface.

Buffer Settings

Undelivered events wait in the Director's persistent queues until their destination acknowledges them. Three settings bound how much can accumulate during an outage. They sit in a collapsed block headed Buffer settings (Optional) at the bottom of the persistent storage area, below the storage type and connection form when the mode has one. Click Show advanced settings to open the fields and Hide advanced settings to close them. The block stays open while one of its fields fails validation.

The block appears for every Director mode when the processing mode is Persistent or Crash-resistant, and not at all for In-memory. Each field is optional: left empty, the Director's own default applies.

Field	Units	Left empty	Description
Retention	Seconds, Hours, Days	48 hours	The ceiling on how long an undelivered event survives a downstream outage. Delivered data is removed as soon as it is acknowledged, so this bounds only what is still waiting. Sets persistent_storage.max_age, in seconds
Queue size limit	Bytes, MB, GB, TB	Each queue's own default	A disk ceiling for each of the Director's three pipeline queues (ingester, router and sender), not a total shared between them. When a queue fills, its oldest waiting events are dropped so delivery keeps running. Each queue's cap is reserved on disk up front, so the caps together have to fit in the free disk. A size set for one queue under persistent_storage.streams in YAML replaces this figure for that queue. Sets persistent_storage.max_bytes, in bytes. Not shown on Orchestrated Directors
Inline payload size	KB, MB	512 KB with Persistent, 1 MB with Crash-resistant	The largest payload carried inside a pipeline bus message. Applies only to a Director that uses a broker as its pipeline bus (see Availability). Sets persistent_storage.payload.max_size, as a suffixed string such as 512kb

Each value must be a whole number. MB, GB and TB are binary multiples: 1 GB is 1,073,741,824 bytes.

Retention and Queue size limit reach the Director in every mode. Left empty, the queue size limit falls back to each queue's shipped default: 1 GB for the sender queue and 512 MB each for the ingester and router queues, scaled down to fit when the Director detects a disk budget. A size set for one queue under persistent_storage.streams in YAML takes precedence over this field for that queue -- see Service Configuration.

Inline payload size has an effect only where the payload rides inside the broker's messages: on an Orchestrated Director, and on a Serverless Director whose connection names a broker address. Keep it at or below the broker's message limit. Managed and Self-managed Directors, and a Serverless Director without a broker address, keep their payloads in the NATS Object Store instead, which has no inline cap: the field is shown and saved for them, but the value is not applied.

note

The helper text under Inline payload size reads 512 KB only for an Orchestrated Director in Persistent mode. A Serverless Director that uses a broker as its pipeline bus also applies 512 KB in Persistent mode when the field is left empty, although its helper text reads 1 MB.

The Review and complete setup step lists these values under a Persistent storage heading, and the Director's Director overview tab shows them as rows. A field left empty shows the default instead of a value: default: 48h, automatic, and default: 512 KB or default: 1 MB.

Editing After Creation

Open the Director from the Directors table and click Manage general settings on the Director overview tab.

The connection fields for the selected backend remain editable. The following are fixed at creation and display as read-only rows:

• Director mode
• Processing mode
• Persistent storage type

Changing any of these requires creating a new Director.

The buffer settings stay editable, with one exception. On a Self-managed Director the fields are shown disabled with their current values, under the note This deployment reads its configuration from disk, so these values are managed there. A Self-managed Director owns its on-disk configuration, so the platform does not push changes to it. This is a separate condition from the fixed-at-creation rows above: change the values in the Director's own configuration file instead, under the persistent_storage keys listed in Service Configuration.