Lookup

Enrich Cribl Compatible

Synopsis

Enriches log events by looking up values in a CSV file and adding corresponding fields to the event. Supports multiple matching modes, case-sensitive/insensitive matching, and various ways to handle multiple matches.

The CSV lookup tables referenced by lookup_file can be created and managed in the Library, or generated on a schedule from a Lookup Source.

Schema

- lookup:
    lookup_file: <string>
    match_mode: <enum>
    match_type: <enum>
    lookup_fields: <LookupField[]>
    output_fields: <OutputField[]>
    ignore_case: <boolean>
    lookup_content: <string>
    lookup_name: <string>
    name: <string>
    skip_lookup: <boolean>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    ignore_missing: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration

The following fields are used to define the processor:

Field	Required	Default	Description
`lookup_file`	Y*	-	Path to a CSV file (with `.csv` extension), or a bare Library entry name (without `.csv` extension, written without spaces) that resolves against the platform lookup library
`lookup_content`	Y*	-	Inline CSV content, header row first, used instead of a file
`lookup_name`	Y*	-	Reuses the table an earlier lookup processor kept under this `name`. See Reusing a Table
`match_mode`	Y	-	Type of matching: `exact`, `regex`, or `cidr`
`match_type`	Y	-	How to handle multiple matches: `first`, `most_specific`, or `all`
`lookup_fields`	Y	-	Array of field mappings used for matching
`output_fields`	Y	-	Array of fields to add to event from matched lookup entries
`ignore_case`	N	`false`	Enable case-insensitive matching
`name`	N	-	Keeps the table this processor loads under this name, for later processors' `lookup_name`. Not a Library entry name: see Reusing a Table
`skip_lookup`	N	`false`	Load the table, and keep it under `name`, without matching the event against it. A processor that sets it needs none of `match_mode`, `match_type`, `lookup_fields` and `output_fields`
`description`	N	-	Explanatory note
`if`	N	-	Condition to run
`ignore_failure`	N	`false`	Continue on lookup errors
`ignore_missing`	N	`false`	Continue if lookup fields don't exist
`on_failure`	N	-	See Handling Failures
`on_success`	N	-	See Handling Success
`tag`	N	-	Identifier
`disabled`	N	`false`	When `true`, the processor is skipped and the event continues to the next one. Lets you take a processor out of the path without removing its configuration

* = one of lookup_content, lookup_file or lookup_name. When more than one is set, the first in that order is used. match_mode, match_type, lookup_fields and output_fields are required unless skip_lookup is true.

Lookup Field

Field	Required	Default	Description
`event_field`	Y	-	Field name in the event
`lookup_field`	N	`event_field`	Corresponding field name in lookup CSV

Output Field

Field	Required	Default	Description
`lookup_field`	Y	-	Field name in lookup CSV
`event_field`	N	`lookup_field`	Field name to add to event
`default_value`	N	-	Value to use if no match found

Match Modes

exact: Exact string matching
regex: Regular expression matching (pattern from lookup table)
cidr: CIDR network matching (for IP addresses)

Match Types

first: Use first matching entry found
most_specific: Use most specific match (primarily for CIDR matching)
all: Use all matching entries (creates array of values)

Details

The processor reads the lookup data from a CSV file where the first row contains header names. It matches event fields against lookup fields and adds corresponding output fields to the event.

note

The processor caches lookup data in memory for performance.
Malformed CSV rows (wrong number of fields) are skipped.
For CIDR matching, more specific networks (longer prefixes) take precedence.
When using match_type: all, matching fields become arrays containing all matched values.

warning

When using regex matching, ensure patterns in the lookup table are valid regular expressions. Invalid patterns will be skipped.
Be cautious with case sensitivity when field names in the CSV differ only by case.
Large CSV files will be kept in memory, so consider memory usage for very large lookup tables.

Reusing a Table

A pipeline that matches several fields against one table can load it once. Give the first lookup processor a name, and have later lookup processors read the same table with lookup_name instead of lookup_file or lookup_content. Set skip_lookup: true on the first one when it should only load the table.

- lookup:
    name: SeverityMap
    lookup_content: |
      level,severity
      1,low
      2,medium
      3,high
    skip_lookup: true
- lookup:
    lookup_name: SeverityMap
    match_mode: exact
    match_type: first
    lookup_fields:
      - event_field: event.level
        lookup_field: level
    output_fields:
      - lookup_field: severity
        event_field: event.severity

lookup_name fails with lookup name '…' not found until the processor that sets name has loaded the table, so that processor must come first.

name is not a Library entry name

The director reads a Library table from lookup_file, never from name. The platform, however, links a lookup processor to a Library table by name when name is set, and by lookup_file only when it is not. A director receives only the Library tables its pipelines are linked to, so a name that differs from a Library lookup_file leaves that table unlinked, unless another processor references it: the director never receives it, the processor fails with lookup file '…' not found, and the table's Assigned resources count leaves the pipeline out. When lookup_file names a Library table, leave name unset or give it the same value.

Examples

Basic IP Lookup

Map IP addresses to locations...

- lookup:
    lookup_file: "ip_locations.csv"
    match_mode: cidr
    match_type: most_specific
    lookup_fields:
      - event_field: source.ip
        lookup_field: network
    output_fields:
      - lookup_field: location
        event_field: source.location
      - lookup_field: datacenter
        event_field: source.datacenter

where ip_locations.csv contains:

network,location,datacenter
10.0.0.0/8,Internal,DC1
192.168.0.0/16,Office,DC2

CIDR Network Zone Mapping

Map IPs to network zones with different security levels...

- lookup:
    lookup_file: "network_zones.csv"
    match_mode: cidr
    match_type: most_specific
    lookup_fields:
      - lookup_field: network
        event_field: client_ip
    output_fields:
      - lookup_field: zone
        event_field: network_zone
      - lookup_field: security_level
        event_field: security

where network_zones.csv contains overlapping networks:

network,zone,security_level
0.0.0/8,internal,low
1.0.0/16,dmz,medium
1.1.0/24,secure,high

An IP address like "10.1.1.50" will match all three networks but return the most specific match (/24), resulting in zone="secure" and security_level="high".

Case-Insensitive Matching

Match values regardless of case...

- lookup:
    lookup_file: "status_map.csv"
    match_mode: exact
    match_type: first
    ignore_case: true
    lookup_fields:
      - event_field: state
        lookup_field: Status
    output_fields:
      - event_field: status_desc
        lookup_field: Description

where status_map.csv contains:

Status,Description
Active,Running
INACTIVE,Stopped

With ignore_case: true, an event with state="ACTIVE" will match the "Active" entry and set status_desc="Running".

Multiple Field Regular Expression Matching

Match on multiple fields using regex patterns...

- lookup:
    lookup_file: "access_rules.csv"
    match_mode: regex
    match_type: all
    lookup_fields:
      - event_field: dept
        lookup_field: department
      - event_field: title
        lookup_field: access_level
    output_fields:
      - event_field: access
        lookup_field: permissions

where access_rules.csv contains:

department,access_level,permissions
IT,.*,admin
Engineering,developer.*,read-write
Engineering,junior.*,read-only

When match_type: all is used with multiple matches, the output field becomes an array containing all matched values.

Default Values for Unmatched Events

Provide default values when no match is found...

- lookup:
    lookup_file: "roles.csv"
    match_mode: exact
    match_type: first
    lookup_fields:
      - event_field: username
        lookup_field: user
    output_fields:
      - event_field: user_role
        lookup_field: role
        default_value: guest

where roles.csv contains:

user,role
alice,admin
bob,user

When looking up a username not in the CSV (like "carol"), the default_value will be used.

Pattern Matching with Message Content

Match log messages against patterns for classification...

- lookup:
    lookup_file: "error_patterns.csv"
    match_mode: regex
    match_type: first
    lookup_fields:
      - event_field: message
        lookup_field: pattern
    output_fields:
      - event_field: alert_level
        lookup_field: severity
      - event_field: response
        lookup_field: action

where error_patterns.csv contains:

pattern,severity,action
error.*,high,alert
warn.*,medium,log
info.*,low,ignore

The processor will match the message content against each pattern and enrich the event with the corresponding severity and action.

Library Entry

Loading a lookup table from the Library by entry name...

- lookup:
    lookup_file: "ip_locations"
    match_mode: cidr
    match_type: most_specific
    lookup_fields:
      - event_field: source.ip
        lookup_field: network
    output_fields:
      - lookup_field: location
        event_field: source.location

where ip_locations is a Library entry name; omitting .csv resolves from the platform lookup library rather than a local file path...

network,location
10.0.0.0/8,Internal
192.168.0.0/16,Office

Synopsis​

Schema​

Configuration​

Lookup Field​

Output Field​

Match Modes​

Match Types​

Details​

Reusing a Table​

Examples​

Basic IP Lookup​

CIDR Network Zone Mapping​

Case-Insensitive Matching​

Multiple Field Regular Expression Matching​

Default Values for Unmatched Events​

Pattern Matching with Message Content​

Library Entry​