Datasets and Profiles: Management

Datasets and profiles are created, configured, and assigned through the DataStream web interface. For conceptual definitions and the type hierarchy, see Overview.

Deletion Constraints

A dataset cannot be deleted if it is assigned to a device or included in a profile. Likewise, a profile cannot be deleted if it is assigned to a device. Remove all associations before deleting. When a deletion is blocked, the UI displays the specific conflicting devices or profiles that must be unassigned first.

A dataset whose filter other datasets use cannot be deleted as it is either. The delete dialog lists those datasets and, once no device or profile still holds the dataset, offers Give each a copy and delete, which gives each of them its own copy of the filter it was using, then deletes the dataset. See Deleting a Dataset Others Follow.

Creating a Dataset

Dataset creation uses a multi-step wizard:

General settings

Enter the dataset name and description.

Dataset configuration

Configure the type-specific collection rules. The configuration interface adapts based on the dataset type — windows, wec, linux, solaris, aix, macos, mssql, mysql, postgres, oracle, snmp, vmware, vcloud, sangfor, openshift, kubernetes, ovirt, proxmox, libvirt, commvault, sentinel, ipam, website, modbus, prometheus, s7, opcua, gnmi, juniper_jti, or elasticsearch_query — and each type also supports an optional preprocessing pipeline assignment. For the collectors available under each type, see Windows Datasets, WEC Datasets, Linux Datasets, Solaris, AIX, and macOS Datasets, Database Datasets, SNMP Datasets, VMware vSphere Datasets, VMware Cloud Director Datasets, Sangfor HCI Datasets, Red Hat OpenShift Datasets, Kubernetes Datasets, oVirt / Red Hat Virtualization Datasets, Proxmox VE Datasets, libvirt Datasets, Commvault Datasets, IPAM Datasets, Service Check Datasets, Elasticsearch Query Datasets, and, for modbus, prometheus, s7, opcua, gnmi, and juniper_jti, the Modbus, Prometheus, Siemens S7, OPC UA, gNMI, and Juniper JTI device pages. File-based log collection (linux_file_log_collector, windows_file_log_collector) is documented on the Linux and Windows Datasets pages.

A dataset type that reads many items of one kind — processes, file systems, services, databases, interfaces — also shows a Filters section below its own fields, which narrows what it collects. See Collection Filters.

Assign to devices

Select one or more devices to assign this dataset to. The device list supports multi-select with search filtering.

Review and create

Review the complete dataset configuration summary before creation. Verify assigned devices and collection rules.

Dataset Detail View

After creation, each dataset has a detail page with three tabs:

General Settings Tab

View and edit the dataset name, description, type, and status (active or passive).

Dataset Configuration Tab

View and edit the type-specific collection rules for this dataset, including its Filters section where the type has one. The read view shows a Filter line summarizing the rule, and on a dataset that other datasets follow, a Used by list naming them. See Collection Filters.

Assigned Devices Tab

View and manage the list of devices assigned to this dataset. Add or remove device assignments.

Collection Filters

A dataset can filter what it collects. A dataset type that reads many items of one kind shows a Filters section in the Dataset configuration step and on the Dataset Configuration tab, below the type's own fields. Types that read a single item, such as CPU or memory usage, have no filter.

The section opens with: "Leave these empty to collect everything this dataset reads. Lists are comma-separated: * matches any run of characters, ? matches one, and case is ignored."

Filter groups

The section holds one group per kind of item the type reads, titled by it — for example Processes, File systems, Disks, Network interfaces, Services, Sites, Databases, Tables, Stored procedures, Containers on the Linux Docker datasets, or Interfaces on SNMP. Network interfaces is both the Linux network interface datasets' group and the Linux WireGuard datasets'.

Each group has an Include … and an Exclude … field, such as Include processes and Exclude processes:

• An item is kept when the include list is empty or one of its patterns matches, and no exclude pattern matches. Exclude wins over include.
• A pattern matches the whole value, not part of it, and * also matches across / and \.
• Leaving both fields empty keeps every item.

Some groups add switches:

Group	Switches
Linux processes	Skip kernel threads
Linux file systems	Skip network file systems, Skip pseudo and container file systems
Linux disks	Skip loop, RAM and optical devices
Linux network interfaces (Network Interfaces and Network Traffic only, not the WireGuard datasets)	Skip container networking interfaces
Services	Only enabled services (Linux), Only services that start automatically (Windows), Skip disabled services (Solaris)
Microsoft SQL databases	Skip system databases, Skip Always On secondaries
Microsoft SQL tables and stored procedures	Include schemas and Exclude schemas; Minimum rows on tables
Linux Docker containers	Include images and Exclude images; Include labels and Exclude labels, each key=value or a key alone. A container is kept only when it passes the names, the images, and the labels.
SNMP interfaces	Only interfaces that are administratively up, Include descriptions, Interface types

New Linux datasets start with three switches on

A new Linux Network Interfaces, Linux File Systems, or Linux Disks dataset starts with Skip container networking interfaces, Skip pseudo and container file systems, or Skip loop, RAM and optical devices switched on, respectively. The switch is visible and can be turned off. Existing datasets never change.

Per-type settings

Some types add settings of their own below the groups. These are never taken from another dataset, even when the filter is.

Dataset	Setting	Default
Microsoft SQL Index Fragmentation	Minimum fragmentation, Minimum pages	10, 100
Microsoft SQL Missing Indexes	Minimum impact	0
Microsoft SQL Blocking Queries	Minimum wait	0
Linux Disk I/O	Whole disks only	off
Commvault Jobs	Job types	every job type
Commvault Events	Minimum severity	information

Using the filter of another dataset

A child type can use the filter of another dataset instead of keeping its own copy. The child types are Busiest Processes, File System Usage, Disk I/O, Network Traffic, every Hyper-V machine dataset, the IIS site and pool datasets, the Microsoft SQL per-database, per-table, and per-procedure datasets, the SNMP per-port and per-tunnel datasets, and the Commvault Jobs and Commvault Events datasets, which can follow a Commvault Clients dataset. Linux WireGuard Peers is a child type too: it can follow a Linux WireGuard Interfaces dataset. So is Linux Docker Logs, which can follow a Linux Docker Containers dataset.

On these, the Filters section starts with Where the filter comes from:

• Set a filter for this dataset — the dataset keeps its own filter, as above.
• Use the filter of another dataset — shows a Dataset picker (Choose a dataset). It offers the active datasets of the types this one may follow, such as a Processes dataset for Busiest Processes. Its helper text reads "Changes to that dataset's filter apply here too."

Once a dataset is chosen:

• Each group the chosen dataset provides turns read-only, shown as "group · from dataset" with a one-line summary and an Open dataset link, which opens in a new tab.
• A group the chosen dataset has no rule for stays editable, with "dataset has no rule for this, so this dataset sets its own."

A follower applies the rule to its own items

A follower applies the rule to the items it reads, not to the other dataset's results. Busiest Processes that follows a Processes dataset keeps the busiest of the processes that the Processes filter would keep.

A dataset can follow a dataset that itself follows another, up to four levels deep, and never in a circle. The console and the web API both refuse any other chain.

Summary, Used by, and editing a followed dataset

• The review step and the detail read view show a Filter line summarizing the rule, for example "Administratively up · with a description · named Gi1/0/*, Te1/1/1 · not *.100". A follower's line reads "Same as dataset (type)".
• The read view of a dataset that others follow lists them under Used by.
• While that dataset is being edited, a notice reads "N datasets use this filter … Your changes apply to them too."

An inventory dataset — one that marks items added, changed, unchanged, or removed — starts its comparison over when its filter or threshold changes. Its next reading marks nothing, rather than marking everything the new filter brings in as added.

Filters need no new permission and no edition: DATASET_CREATE and DATASET_EDIT govern them as before.

Deleting a Dataset Others Follow

A dataset whose filter other datasets use is protected from deletion. The checks run in this order: devices, then profiles, then followers, so a dataset with devices assigned still reports the devices first.

When followers are all that remain, the delete modal names them under N datasets use this filter, and its button reads Give each a copy and delete. The copy is exact: each follower keeps collecting the same items, and after the delete it holds the filter as its own.

In the public API, DELETE /datasets/{id} answers 409 DATASET_HAS_FOLLOWERS for a followed dataset. ?detachFollowers=true copies the filter to each follower and deletes, in one transaction. DATASET_DELETE governs it as before.

Configuration keys

In a definition, the filter sits under properties.filter:

Key	Description
filter.<group>	One object per group, such as filter.process, filter.filesystem, filter.database, or filter.container, each holding include and exclude lists and the group's switches (skip_kernel_threads, skip_virtual, …) or extra lists (images, exclude_images, labels, exclude_labels on container).
filter.follow	The id of the dataset whose filter this one uses, as a string. Groups set beside it are this dataset's own and apply only where the followed dataset has no rule for that group: where both set one, the followed dataset's wins.

The per-type settings are properties of their own, beside filter: min_fragmentation_pct, min_pages, min_impact_pct, min_wait_sec, whole_disks_only, job_types, and min_severity.

A Processes filter that leaves out container shims and kernel threads...	properties: interval: 3600 filter: process: exclude: ["containerd-shim", "pause"] skip_kernel_threads: true
A dataset that follows another dataset's filter, and adds a rule of its own...	properties: interval: 300 filter: follow: "7318842175993245696" table: min_rows: 1000

Dataset Operations

• Clone: Create a copy of an existing dataset with all its configuration. The cloned dataset requires a new name and can be modified independently.
• Export template: Open the dataset as a Bicep template, to deploy again or to change first. See Export Template.
• Delete: Remove a dataset. A confirmation modal displays before deletion to prevent accidental removal. When other datasets use this dataset's filter and no device or profile still holds it, the modal names them under N datasets use this filter, and its button reads Give each a copy and delete.

Creating a Profile

Profile creation uses a multi-step wizard. Profiles are created with active status by default.

Define Profile

Enter the profile name and description.

Select Datasets

Select one or more existing datasets to include in this profile. The dataset list supports multi-select with filtering.

Assign to Devices

Select one or more devices to assign this profile to. Device assignment is optional and can be configured later.

Review and Create Profile

Review the profile summary including selected datasets and assigned devices before creation.

Profile Detail View

The profile detail page has three tabs: General Settings, Profile Configuration, and Assigned Devices — providing access to the profile's settings, its dataset composition, and its device assignments.

The detail page's Actions menu offers Export template, which opens the profile as a Bicep template, to deploy again or to change first. See Export Template.

Device Integration

Datasets connect to devices through the Data collection step of the device creation wizard, or the Data collection tab on the device's detail page. When configuring a device's data collection:

1. A selection drawer displays available datasets and profiles
2. Select one or more datasets or profiles to assign
3. A confirmation modal with a switch control confirms the assignment change
4. The device begins collecting data according to the assigned dataset rules

Exclusive Assignment

A device can be assigned either datasets or a profile, not both. Assigning one type replaces any existing assignment of the other type.

Each device tracks its configuration mode (dataset or profile), determining whether it receives collection rules from individual datasets or from a profile.

Assigned datasets appear in the device's detail view under the Data collection tab (see Devices Management) and can be managed from either the device or dataset side of the relationship.