Skip to main content

Regex Replace

Parse Pattern Matching Data Transformation

Synopsis​

A text processing processor that finds and replaces text patterns using regular expressions, for data cleaning, formatting, and normalization.

Schema​

- regex_replace:
    field: <ident>
    regex: <string>
    replacement: <string>
    target_field: <ident>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    ignore_missing: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration​

The following fields are used to define the processor:

FieldRequiredDefaultDescription
fieldY-Field containing the text to process
regexY-Regular expression pattern to match
replacementY-Replacement text or pattern
target_fieldNfieldField to store the modified text
descriptionN-Explanatory note
ifN-Condition to run
ignore_failureNfalseContinue if regex processing fails
ignore_missingNfalseContinue if source field doesn't exist
on_failureN-See Handling Failures
on_successN-See Handling Success
tagN-Identifier
disabledNfalseWhen true, the processor is skipped and the event continues to the next one. Lets you take a processor out of the path without removing its configuration

Details​

The processor uses regular expressions to find and replace text patterns within string fields. It supports both simple text replacement and pattern matching with capture groups, which the replacement refers to as $1, $2 or ${name}.

note

This processor is an alias for the gsub processor, providing the same functionality with a more descriptive name.

Regular expression patterns support character classes, quantifiers, anchors, and grouping. The replacement string can include backreferences ($1, $2, etc.) to captured groups from the regex pattern.

The processor handles all occurrences of the pattern within the text, making it suitable for comprehensive text cleaning and transformation tasks.

warning

Test regex patterns thoroughly to avoid unintended matches or performance issues with complex patterns.

Regular Expression Semantics​

Patterns use RE2 syntax, with lookahead ((?=…), (?!…)) and lookbehind ((?<=…), (?<!…)) added. Backreferences (\1, \k<name>), atomic groups and possessive quantifiers are not supported: a pattern that uses one fails to compile, and the processor fails the record — its on_failure processors run, and ignore_failure applies. A lookbehind cannot contain a capture group or another lookaround. \d, \w and \s match ASCII characters.

Matching has no per-match timeout, because none is needed: no pattern can make matching backtrack without bound.

Matching is unanchored and case-sensitive. A plain string matches anywhere in the field — anchor with ^ and $, and prefix the expression with (?i) for case-insensitive matching.

A pattern is rejected before it compiles when it:

  • exceeds 1000 characters,
  • uses more than 100 quantifiers (*, +, {n,m}),
  • nests groups more than 10 levels deep, or
  • takes longer than 100ms to compile.

Examples​

Basic Text Replacement​

Replacing simple text patterns...

{
  "message": "Error: Failed to connect to server"
}
- regex_replace:
    field: message
    regex: "Error:"
    replacement: "WARNING:"

updates the error level:

{
  "message": "WARNING: Failed to connect to server"
}

Pattern Matching with Capture Groups​

Using capture groups for reformatting...

{
  "timestamp": "2024-01-15 14:30:25"
}
- regex_replace:
    field: timestamp
    regex: '(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})'
    replacement: "$2/$3/$1 $4:$5:$6"
    target_field: formatted_timestamp

reformats the date:

{
  "timestamp": "2024-01-15 14:30:25",
  "formatted_timestamp": "01/15/2024 14:30:25"
}

Email Masking​

Masking email addresses for privacy...

{
  "user_info": "Contact john.doe@example.com for support"
}
- regex_replace:
    field: user_info
    regex: "([a-zA-Z0-9._%+-]+)@([a-zA-Z0-9.-]+\\.[a-zA-Z]{2,})"
    replacement: "***@$2"

masks the username portion:

{
  "user_info": "Contact ***@example.com for support"
}

Log Level Normalization​

Normalizing various log level formats...

{
  "log_line": "[ERR] Database connection failed",
  "log_line2": "WARN: Memory usage high"
}
- regex_replace:
    field: log_line
    regex: "\\[(ERR|ERROR)\\]"
    replacement: "[ERROR]"
- regex_replace:
    field: log_line2
    regex: "WARN:"
    replacement: "[WARNING]"

standardizes log levels:

{
  "log_line": "[ERROR] Database connection failed",
  "log_line2": "[WARNING] Memory usage high"
}

URL Path Extraction​

Extracting paths from URLs...

{
  "request_url": "https://api.example.com/v1/users/123?param=value"
}
- regex_replace:
    field: request_url
    regex: "https?://[^/]+(/[^?]*)"
    replacement: "$1"
    target_field: url_path

extracts just the path:

{
  "request_url": "https://api.example.com/v1/users/123?param=value",
  "url_path": "/v1/users/123"
}

Multi-Pattern Replacement​

Applying multiple regex replacements...

{
  "raw_text": "User ID: 12345, Phone: (555) 123-4567, Email: user@domain.com"
}
- regex_replace:
    field: raw_text
    regex: "\\d{5}"
    replacement: "XXXXX"
- regex_replace:
    field: raw_text
    regex: "\\(\\d{3}\\) \\d{3}-\\d{4}"
    replacement: "XXX-XXX-XXXX"
- regex_replace:
    field: raw_text
    regex: "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}"
    replacement: "***@***.***"

sanitizes sensitive information:

{
  "raw_text": "User ID: XXXXX, Phone: XXX-XXX-XXXX, Email: ***@***.***"
}