Skip to main content
Version: 1.3.0

Coalesce

Transform Field Selection

Returns the first non-null, non-empty value from a list of fields, similar to SQL COALESCE function.

Schema​

- coalesce:
    fields: <ident[]>
    target_field: <ident>
    ignore_empty_string: <boolean>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    ignore_missing: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration​

The following fields are used to define the processor:

FieldRequiredDefaultDescription
fieldsY-Array of field names to check in order
target_fieldY-Target field to store the first non-null value
ignore_empty_stringNfalseSkip empty strings when finding first value
descriptionN-Explanatory note
ifN-Condition to run processor
ignore_failureNfalseContinue if processor fails
ignore_missingNfalseContinue if all fields are missing
on_failureN-Processors to run on failure
on_successN-Processors to run on success
tagN-Processor identifier

Details​

The processor skips null values automatically and by default treats empty strings as valid values, though this can be changed using ignore_empty_string: true to skip them as well. Fields are evaluated in the order specified in the array, with missing fields automatically skipped.

The processor preserves the original data type of the selected value and fails if all fields are missing or empty unless ignore_missing: true is configured.

This processor is particularly useful for field prioritization when multiple options are available, providing fallback values when primary fields might be missing. It excels at data consolidation by merging similar fields from different log sources, such as finding the best contact email from multiple possibilities or choosing the most appropriate client IP from various network fields.

The processor also helps with message standardization by selecting the best message field from different formats across diverse log sources.

Examples​

Basic Usage​

Finding first non-null value from user fields...

{
  "user": {
    "name": null,
    "username": "john_doe",
    "id": "12345"
  }
}
- coalesce:
    fields: ["user.name", "user.username", "user.id"]
    target_field: display_name

selects first available value:

{
  "user": {
    "name": null,
    "username": "john_doe",
    "id": "12345"
  },
  "display_name": "john_doe"
}

Email Fallback​

Default behavior treats empty strings as valid values...

{
  "user": {
    "primary_email": "",
    "work_email": "[email protected]",
    "personal_email": "[email protected]"
  }
}
- coalesce:
    fields: ["user.primary_email", "user.work_email", "user.personal_email"]
    target_field: contact_email

returns first value (empty string):

{
  "user": {
    "primary_email": "",
    "work_email": "[email protected]", 
    "personal_email": "[email protected]"
  },
  "contact_email": ""
}

Empty Strings​

Using ignore_empty_string to skip empty values...

{
  "user": {
    "primary_email": "",
    "work_email": "[email protected]",
    "personal_email": "[email protected]"
  }
}
- coalesce:
    fields: ["user.primary_email", "user.work_email", "user.personal_email"]
    target_field: contact_email
    ignore_empty_string: true

skips empty string, finds next value:

{
  "user": {
    "primary_email": "",
    "work_email": "[email protected]",
    "personal_email": "[email protected]"
  },
  "contact_email": "[email protected]"
}

IP Resolution​

Selecting best available IP address from multiple sources...

{
  "network": {
    "client_ip": null,
    "forwarded_ip": "",
    "remote_ip": "192.168.1.100"
  },
  "source": {
    "ip": "10.0.0.50"
  }
}
- coalesce:
    fields: ["network.client_ip", "network.forwarded_ip", "network.remote_ip", "source.ip"]
    target_field: client_address
    ignore_empty_string: true

skips null/empty, finds first valid IP:

{
  "network": {
    "client_ip": null,
    "forwarded_ip": "",
    "remote_ip": "192.168.1.100"
  },
  "source": {
    "ip": "10.0.0.50"
  },
  "client_address": "192.168.1.100"
}

Message Fallback​

Finding message from various log format fields...

{
  "event": {
    "original": null
  },
  "message": null,
  "log": {
    "message": "User login successful"
  },
  "raw_message": "Login event"
}
- coalesce:
    fields: ["event.original", "message", "log.message", "raw_message"]
    target_field: processed_message

selects first available message:

{
  "event": {
    "original": null
  },
  "message": null,
  "log": {
    "message": "User login successful"
  },
  "raw_message": "Login event",
  "processed_message": "User login successful"
}

Timestamps​

Selecting best timestamp from multiple time fields...

{
  "@timestamp": null,
  "event": {
    "created": "2023-12-01T10:30:00Z"
  },
  "log": {
    "timestamp": "2023-12-01T10:29:55Z"
  },
  "received_at": "2023-12-01T10:30:05Z"
}
- coalesce:
    fields: ["@timestamp", "event.created", "log.timestamp", "received_at"]
    target_field: event_time

chooses first non-null timestamp:

{
  "@timestamp": null,
  "event": {
    "created": "2023-12-01T10:30:00Z"
  },
  "log": {
    "timestamp": "2023-12-01T10:29:55Z"
  },
  "received_at": "2023-12-01T10:30:05Z",
  "event_time": "2023-12-01T10:30:00Z"
}