Regex Extract

Parse Cribl Compatible

Synopsis

Extracts named fields from text using regular expressions with named capture groups.

Schema

- regex_extract:
    field: <ident>
    regex: <string>
    lang: <enum>
    target_field: <ident>
    additional_regex: <string[]>
    description: <text>
    field_name_format: <string>
    filter: <script>
    if: <script>
    ignore_failure: <boolean>
    ignore_missing: <boolean>
    max_exec: <integer>
    on_failure: <processor[]>
    on_success: <processor[]>
    overwrite_existing: <boolean>
    tag: <string>

Configuration

The following fields are used to define the processor:

Field	Required	Default	Description
field	N	message	Field containing text to extract from. Under lang: js the default is _raw
regex	Y	-	Regular expression with named capture groups. Under lang: js, parsed as a /pattern/flags literal, where the g flag scans for all matches rather than only the first
lang	N	-	Regex dialect. Empty selects the native engine; js (javascript, ecmascript) selects the ECMAScript-compatible engine used for Cribl pipelines
filter	N	-	Cribl-style JavaScript truthiness expression evaluated after if. A falsy result silently skips the processor. Distinct from if, which uses the native expression language
target_field	N	-	Field to store extracted values when no named groups are used. A bare $Table value instead switches on column routing — see below
additional_regex	N	-	Additional patterns to match after primary regex
description	N	-	Explanatory note
field_name_format	N	-	Template for formatting extracted field names (${name})
if	N	-	Condition to run
ignore_failure	N	false	Continue on regex match failures
ignore_missing	N	false	Continue if source field doesn't exist
max_exec	N	100	Maximum number of matches to process
on_failure	N	-	See Handling Failures
on_success	N	-	See Handling Success
overwrite_existing	N	false	Replace existing fields instead of converting to array
tag	N	-	Identifier
disabled	N	false	When true, the processor is skipped and the event continues to the next one. Lets you take a processor out of the path without removing its configuration

Details

The processor supports dynamic field naming using _NAME_ and _VALUE_ pattern pairs, field name formatting, and handling of multiple matches.

Golang regular expressions provied named capture groups to extract fields.

warning

Complex regular expressions on large texts may impact performance

Each named group becomes a field in the output. Special _NAME_n and _VALUE_n pairs allow dynamic field naming based on extracted content.

Routing Groups to a Virtual Table

Setting target_field to a bare $Table name — no dot, no field path — switches the processor into column-routing mode: every named group is written to the same-named column of that virtual table instead of to the event map.

- regex_extract:
    field: message
    regex: "(?<Object>[^ ]+) (?<Operation>[^ ]+)"
    target_field: $ASimAuditEventLogs

The prefix lives on the processor because a Go capture-group name cannot contain $. A group whose name is not a column of that table fails loudly rather than being dropped — group names are written by the pipeline author, so a mismatch is an authoring error, not wire noise.

In this mode target_field is not a container: the unnamed-group behaviour above does not apply, and nothing is written to a field called $Table.

note

The _NAME_n and _VALUE_n pairs must use matching indices, e.g. _NAME_0 with _VALUE_0

Multiple regex patterns, array conversion for duplicate fields, field name templating, and match count limiting are also supported.

Field names are automatically sanitized to remove invalid characters. However, the field_name_format should produce valid field names. Also, when overwrite_existing is set to false, duplicate matches are converted to arrays.

warning

Be careful with the max_exec setting when dealing with high-frequency matches.

Consider using ignore_failure when regex patterns might not match all inputs.

Regular Expression Semantics

Patterns use RE2 syntax, with lookahead ((?=…), (?!…)) and lookbehind ((?<=…), (?<!…)) added. Backreferences (\1, \k<name>), atomic groups and possessive quantifiers are not supported: a pattern that uses one fails to compile, and the processor fails the record — its on_failure processors run, and ignore_failure applies. A lookbehind cannot contain a capture group or another lookaround. \d, \w and \s match ASCII characters.

Matching has no per-match timeout, because none is needed: no pattern can make matching backtrack without bound.

Matching is unanchored and case-sensitive. A plain string matches anywhere in the field — anchor with ^ and $, and prefix the expression with (?i) for case-insensitive matching.

A pattern is rejected before it compiles when it:

• exceeds 1000 characters,
• uses more than 100 quantifiers (*, +, {n,m}),
• nests groups more than 10 levels deep, or
• takes longer than 100ms to compile.

These semantics describe the native engine. Under lang: js the pattern is parsed as a /pattern/flags literal with JavaScript's syntax and matched by the ECMAScript-compatible engine instead. Two constructs JavaScript has are not supported there: backreferences, and a lookbehind that contains a capture group. Capture groups are numbered as JavaScript numbers them, named or not, left to right.

Examples

Basic

Extracting a numeric value with a static field name...	{ "message": "metric1=23 metric2=42" } - regex_extract: field: message regex: "metric1=(?<metric1>\\d+)"
creates a new field:	{ "message": "metric1=23 metric2=42", "metric1": "23" }

Complex Logs

Extracting multiple fields from structured log...

{ "message": "462559d4a487[471]: 172.23.0.6 - - [26/Feb/2024:20:22:38 +0000] \"GET /path HTTP/1.1\" 200 87533" } - regex_extract: field: message regex: "^(?<container_id>[^\\s]+)\\[(?<process_id>\\d+)\\]:\\s+(?<remote_host>[^\\s]+)\\s+(?<remote_user>-)\\s+(?<auth_user>-)\\s+\\[(?<timestamp>[^\\]]+)\\]\\s+\"(?<request_method>\\w+)\\s+(?<requested_url>[^\\s]+)\\s+(?<http_version>[^\"]+)\"\\s+(?<status>\\d+)\\s+(?<bytes>.+)$"

yields HTTP log components:

{ "message": "462559d4a487[471]: 172.23.0.6...", "container_id": "462559d4a487", "process_id": "471", "remote_host": "172.23.0.6", "request_method": "GET", "requested_url": "/path", "http_version": "HTTP/1.1", "status": "200", "bytes": "87533" }

Dynamic Fields

Extracting key-value pairs as dynamic fields...	{ "message": "name=\"John Doe\" age=30 email=\"john@example.com\"" } - regex_extract: field: message regex: "(?<_NAME_0>[^\\s=]+)=\"?(?<_VALUE_0>(?<=\")[^\"]*|[^\\s\"]+)"
creates new fields based on the extracted names:	{ "message": "name=\"John Doe\" age=30 email=\"john@example.com\"", "name": "John Doe", "age": "30", "email": "john@example.com" }

Formatting

Formatting extracted field names...	{ "message": "key=value" } - regex_extract: field: message regex: "(?<_NAME_0>[^=]+)=(?<_VALUE_0>.+)" field_name_format: "${name}_field"
adds suffixes:	{ "message": "key=value", "key_field": "value" }

Multi-Match

Extracting multiple matches with array conversion...	{ "message": "value=1 value=2 value=3 value=4 value=5" } - regex_extract: field: message regex: "value=(?<value>\\d+)" max_exec: 3
creates an array of up to max_exec matches:	{ "message": "value=1 value=2 value=3 value=4 value=5", "value": ["1", "2", "3"] }

Structured Data

Using multiple regexes with structured data...	{ "message": "<134>1 2020-12-22T17:06:08Z CORP_INT_NLB CheckPoint 18160 - [action:\"Accept\"; conn_direction:\"Internal\"]" } - regex_extract: field: message regex: "\\[(?<__fields>.*?)\\]" additional_regex: - "(?<_NAME_0>[^:]+):\"(?<_VALUE_0>[^\"]+)\""
extracts nested key-value pairs. __fields is an ordinary named group, so it is written to the event too — remove it with Remove if it is not wanted:	{ "message": "<134>1 2020-12-22T17:06:08Z...", "__fields": "action:\"Accept\"; conn_direction:\"Internal\"", "action": "Accept", "conn_direction": "Internal" }