Skip to main content

Foreach

Control Flow Elastic Compatible

Synopsis​

Runs a processor on each element of an array or object when the number of elements is unknown.

Schema​

foreach:
  - field: <ident>
  - processor: <processor>
  - description: <text>
  - if: <script>
  - ignore_failure: <boolean>
  - ignore_missing: <boolean>
  - on_failure: <processor[]>
  - on_success: <processor[]>
  - tag: <string>

Configuration​

FieldRequiredDefaultDescription
fieldY-Field containing the array or object to iterate over
processorY-The processor configuration to run on each element
descriptionN-Explanatory note for documentation
ifN-Conditional expression determining if the processor should run
ignore_failureNfalseIf true, individual element processing failures are ignored
ignore_missingNfalseIf true and field does not exist, exits quietly without modifications
on_failureN-Processors to run if an error occurs during iteration
on_successN-Processors to run after successful iteration completion
tagN-Identifier for logging and debugging

Details​

The processor supports both arrays and maps (objects) as input, and can execute any valid processor on each element. It uses the following procedure:

  • it first locates the specified field in the document, and determines if the field contains an array or a map

  • if an array of a map exists, it goes through each element storing the current value in _ingest._value, the current key/index in _ingest._key, executing the specified processor, and capturing any errors for failure handling.

  • finally, it cleans up any temporary _ingest fields, and runs the success/failure processors as appropriate.

warning

Be careful when modifying the _ingest fields directly as they are used internally by the processor.

The processor provides several error handling mechanisms:

ignore_failure
Skip individual element failures
ignore_missing
Skip missing input fields
on_failure
Execute recovery processors on error
on_success
Execute follow-up processors on success

If an error occurs—and neither ignore_failure nor on_failure was specified—the processor leaves the already processed elements modified, stops processing the remaining elements, and returns the error.

Examples​

Arrays​

Process each array element...

{
  "messages": [
    {"text": "hello", "read": false},
    {"text": "world", "read": false}
  ]
}
foreach:
  - field: messages
  - processor:
      set:
        - field: "_ingest._value.read"
        - value: true

and mark each as read:

{
  "messages": [
    {"text": "hello", "read": true},
    {"text": "world", "read": true}
  ]
}

Maps​

Process each key-value pair...

{
  "counts": {
    "errors": "5",
    "warnings": "3",
    "info": "10"
  }
}
foreach:
  - field: counts
  - processor:
      convert:
        - field: "_ingest._value"
        - type: integer

converting string values to integers:

{
  "counts": {
    "errors": 5,
    "warnings": 3,
    "info": 10
  }
}

Nested​

Chain multiple processors for each element...

{
  "users": [
    {"name": "john doe", "role": "admin"},
    {"name": "jane smith", "role": "user"}
  ]
}
foreach:
  - field: users
  - processor:
      pipeline:
        - uppercase:
            - field: "_ingest._value.name"
        - set:
            - field: "_ingest._value.processed"
            - value: true

applying multiple transformations:

{
  "users": [
    {"name": "JOHN DOE", "role": "admin", "processed": true},
    {"name": "JANE SMITH", "role": "user", "processed": true}
  ]
}

Error Handling​

Handle processing failures gracefully...

{
  "values": [
    {"num": "123"},
    {"num": "invalid"},
    {"num": "456"}
  ]
}
foreach:
  - field: values
  - processor:
      convert:
        - field: "_ingest._value.num"
        - type: integer
  - ignore_failure: true
  - on_failure:
      - set:
          - field: "_ingest._value.error"
          - value: "Conversion failed"

preserving data and adding error info:

{
  "values": [
    {"num": 123},
    {"num": "invalid", "error": "Conversion failed"},
    {"num": 456}
  ]
}