Library

Standard and Enterprise

Synopsis

The Library is where you create and manage the reusable assets that pipeline processors reference: lookup tables, schemas, and grok patterns. Defining these assets once in the Library lets multiple pipelines share them instead of embedding the same data or definitions in every configuration.

Each asset type maps to a processor that consumes it:

• Lookup tables provide the CSV data used by the Lookup and Enrich processors.
• Schemas define the structures validated by the Check Schema processor.
• Grok patterns supply the extraction patterns used by the Grok processor.

The Library also holds samples — reusable log data that device simulation, the pipeline debugger and prediction draw on rather than asking you to paste the same records in again. Samples are covered separately under Library: Samples.

This page covers managing these assets through the web interface. For how each asset is referenced inside a pipeline, see the linked processor pages.

Director version requirement

The Library requires Director version 2.0 or later. If any director or director cluster in your fleet runs an earlier version, an alert at the top of the Library page lists the affected instances. Upgrade those directors to use Library assets in their pipelines.

Accessing the Library

To open the Library:

1. Click the hamburger menu in the top left corner.
2. Select Content Management > Library.

The page opens with six tabs: Lookup Tables (default), Schemas, Grok Patterns, Scripts, Samples, and SNMP Vendors. Each tab lists the assets of that type and provides its own create flow. Every tab except SNMP Vendors also has detail pages.

This page covers the first three, and SNMP Vendors in SNMP Vendors. Samples has a page of its own — see Library: Samples — and Scripts is not covered here. To run a Library script from a pipeline, and for what signing it changes, see the Script processor.

Working with Library Assets

The three asset types share a common interaction model. SNMP Vendors does not: it has no detail page and no Assigned resources column, so none of the following applies to it.

• Assigned resources - Every overview table includes an Assigned resources column showing how many resources currently use the asset. Click the count to list the referencing pipelines (for schemas, referencing targets are listed as well where applicable).
• Drawer - Click an asset's name in the table to open a detail drawer on the right side, summarizing the asset and offering quick actions.
• Clone - Cloning opens the create flow pre-filled with the source asset's values, so you can save it under a new name.
• Delete - An asset that is assigned to one or more resources cannot be deleted. A blocking modal lists the assigned resources; remove the asset from them first, then delete it.
• Activity Logs - Each asset's detail page includes an Activity Logs tab with a searchable audit trail of actions performed on that asset.

Names with spaces

A director finds a Library entry by its name with every space removed: a lookup table named Geo IP ships to the director as GeoIP.csv, and a pipeline reaches it as GeoIP. Reference a lookup table, grok pattern, schema or script by its name without spaces. A reference that keeps the spaces still links the entry, but the processor on the director looks for a name it does not have and fails, silently when ignore_failure is set. Names stay case-sensitive; only spaces are removed.

Two entries of the same type cannot have names that differ only by spaces, such as Geo IP and GeoIP, because they would ship to the same file on a director. Creating or renaming an entry onto such a name is refused as a name already in use. Pairs created before this check keep working as before: a reference that names one of them exactly links that one, and a reference that matches neither exactly, such as GeoIP for Geo IP and Ge oIP, links and ships neither, and the platform logs a warning. Rename one of them.

Lookup files given as a path ending in .csv, built-in schema models and inline schema definitions are not affected.

The sections below cover the specifics of each asset type.

Lookup Tables

A lookup table is a CSV dataset that pipelines match against to enrich or transform events.

A lookup table can also be generated on a schedule from a recipe — a script, a SQL query, or an HTTP endpoint — instead of being uploaded or entered by hand. See Lookup Sources.

Lookup Tables Table

The table lists each lookup table with the following columns:

• File Name - The lookup table's name. Click to open the detail drawer.
• Type - The storage method, Memory or Disk.
• Size - The size of the stored data.
• Assigned resources - Count of resources referencing this table.

Above the table, the Search field filters by name, the Type dropdown filters by storage method (All, Memory, or Disk), and the Create new lookup table button opens the create flow. Pagination controls appear below the table.

The row action menu (the vertical ellipsis on each row) offers Manage lookup table, Clone lookup table, Download lookup table (exports the data as a CSV file), and Delete lookup table. The detail drawer offers Manage lookup table and Clone lookup table as quick actions.

Create a Lookup Table

Click Create new lookup table and complete three steps:

Lookup Table Details

• Lookup table name - Required. Between 3 and 64 characters.
• Description - Optional.
• Choose storage method - Required. Select Store in memory (recommended for smaller datasets that need fast access during execution) or Store on disk (recommended for larger datasets that do not need to be fully loaded into memory).

Lookup Table Configuration

Choose how to populate the table:

• Upload CSV file - Upload a CSV file. Only the CSV format is accepted, with a maximum file size of 10 MB. A different format produces an error and blocks the next step.
• Enter manually - Enter the data directly in a Monaco editor in CSV format. The first row holds the column headers, and values are comma-separated. The content is validated as CSV.

Review and Complete Setup

Review the summary of the previous steps, returning to any step to make changes, then complete the setup.

Lookup Table Details

The detail page has three tabs:

• General Overview - Displays the Name, Description, Type, Size, and the created and last-updated timestamps. Editable fields are saved from this tab.
• Lookup Table Configuration - Shows the data and lets you edit it in two interchangeable views. Table view supports adding and deleting rows; Text view shows the data as CSV in a Monaco editor. Switching views warns that unsaved changes in the current view will be lost. The Replace file action overwrites the existing content with a newly uploaded CSV file.
• Activity Logs - A searchable record of actions performed on the table.

The page header also provides an Actions menu containing Clone lookup table and Delete lookup table.

Schemas

A schema defines a data structure that the Check Schema processor validates events against.

Schemas Table

The table lists each schema with the following columns:

• Schema Name - The schema's name. Click to open the detail drawer.
• Description - The schema's description.
• Version - An optional version label.
• Format - The schema's format: JSON, Parquet or Avro.
• Assigned resources - Count of resources referencing this schema.

The platform's own table schemas — the Microsoft Sentinel and Defender XDR tables, and the OCSF classes — are all Parquet. A schema you create can be any of the three.

Above the table, the Search field filters by name, the Origin filter narrows the list to All, Platform, Custom or Hidden schemas, and the Create new schema button opens the create flow. Unlike Lookup Tables, the Schemas tab has no Type filter. Pagination controls appear below the table.

The detail drawer and the row action menu offer Manage schema and Clone schema. The destructive action depends on where the schema came from: a schema you created offers Delete schema, while a platform schema offers Hide schema instead, since the platform's own schemas cannot be deleted. A hidden schema offers Restore schema, and a platform schema you have edited also offers Reset to platform, which replaces your changes with the platform's current version under the same name.

Create a Schema

Click Create new schema and complete three steps:

Schema Details

• Schema name - Required. Between 3 and 64 characters.
• Version - Optional. A label to track changes for your own reference.
• Description - Optional.

Schema Configuration

• Schema Format - Required. Select JSON, Parquet, or Avro. The format cannot be changed after creation.
• Enter the schema definition in a Monaco editor. The definition is validated against the selected format. The JSON definition syntax is documented under Avro and Parquet.

A registered schema is referenced by name from the Enforce Schema and Check Schema processors.

Review and Complete Setup

Review the summary of the previous steps, returning to any step to make changes, then complete the setup.

Schema Details

The detail page has three tabs:

• General Overview - Displays and lets you edit the Name, Description, and Version, along with the created and last-updated timestamps. Changing the name shows a warning that assigned resources must be updated manually.
• Schema Configuration - Shows the Schema Format (read-only) and the Schema Definition (editable).
• Activity Logs - A searchable record of actions performed on the schema.

Grok Patterns

A grok pattern is a reusable extraction pattern that the Grok processor applies to pull structured fields out of unstructured text.

Grok Patterns Table

The table lists each grok pattern with the following columns:

• File Name - The pattern's name. Click to open the detail drawer.
• Description - The pattern's description.
• Assigned resources - Count of resources referencing this pattern.

Above the table, the Search field filters by name, the Origin filter narrows the list to All, Platform, Custom or Hidden patterns, and the Create new Grok pattern button opens the create flow. Pagination controls appear below the table.

The detail drawer and the row action menu offer Manage pattern and Clone pattern. As with schemas, the destructive action depends on where the pattern came from: a pattern you created offers Delete pattern, while a platform pattern offers Hide pattern instead, since the platform's own patterns cannot be deleted. A hidden pattern offers Restore pattern, and a platform pattern you have edited also offers Reset to platform, which replaces your changes with the platform's current version under the same name.

Create a Grok Pattern

Click Create new Grok pattern and complete three steps:

Grok Pattern Details

• File name - Required. Between 3 and 64 characters.
• Description - Optional.

Grok Pattern Configuration

Enter the pattern content in a Monaco editor, for example:

%{IP:client_ip} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:status}

Review and Complete Setup

Review the summary of the previous steps, returning to any step to make changes, then complete the setup.

Grok Pattern Details

The detail page has three tabs:

• General Overview - Displays and lets you edit the Name and Description, along with the created and last-updated timestamps. Changing the name shows a warning that assigned resources must be updated manually.
• Grok Pattern Configuration - Shows and lets you edit the Grok Pattern Content.
• Activity Logs - A searchable record of actions performed on the pattern.

SNMP Vendors

An SNMP device is identified by the sysObjectID it reports, against a built-in list of about 8,900 devices, and otherwise by its sysDescr. The SNMP Vendors tab shows that list and lets you add a sysObjectID, correct one, or hide one so that devices reporting it are identified by their sysDescr instead.

The tab is available on the Standard and Enterprise editions. Without it, the built-in list still identifies devices, and a device's Vendor can still be chosen by hand.

SNMP Vendors Table

The table lists each sysObjectID with the following columns:

• sysObjectID - The value the device reports. Rows you changed carry a tag: Added, Modified, or Hidden. Built-in rows carry none.
• Vendor - The manufacturer.
• Model - The model, where one is known.
• Device class - What kind of device it is.

Above the table, the search field (Search by sysObjectID, vendor or model) filters the list, the Source filter narrows it to All, Built-in, Added, Modified, or Hidden rows, the Class filter narrows it by device class, and the Create new SNMP vendor button opens the create dialog.

The row menu offers only what applies to the row and what you may do:

Row	Actions	Needs
Built-in, untouched	Edit (creates a change) and Hide	Library create
Added	Edit / Delete	Library edit / Library delete
Modified	Edit and Hide / Reset to built-in	Library edit / Library delete
Hidden	Restore	Library delete

Create or Edit an Entry

Create new SNMP vendor and Edit open the same dialog:

• sysObjectID - Required. Fixed once the entry exists.
• Vendor - Required. At most 64 characters. Records carry it in observer.vendor. A vendor that names a manufacturer the Director has datasets for (Cisco, F5, Fortinet) lets those datasets read the device.
• Device class (optional) - Networking, Computer, Storage, Printer, Phone, Power, KVM, Video, or Software. Records carry it in observer.type.
• Model (optional) - At most 128 characters. Records carry it in observer.product.

The SNMP device's connection check opens the same dialog from Add to SNMP vendors, for a device whose manufacturer is unknown.

Limits and Delivery

• A tenant keeps up to 5,000 changes.
• Changes go at once to the tenant's Directors on v2.2.3 or later as library/snmp_vendor/vendors.json, and a running device picks them up at its next identification.
• A standalone Director reads the same file from config/library/snmp_vendor/vendors.json:

{"entries": [
  {"object_id": ".1.3.6.1.4.1.99999.7", "vendor": "Acme", "device_class": "NETWORKING", "model": "Edge 7"},
  {"object_id": ".1.3.6.1.4.1.9.1.1208", "hidden": true}
]}