Skip to main content

Date

Transform Elastic Compatible

Synopsis​

Parses dates from a field in various formats into a standardized timestamp.

Schema​

date:
  - field: <ident>
  - formats: <enum[]>
  - description: <text>
  - if: <script>
  - ignore_failure: <boolean>
  - locale: <string>
  - on_failure: <processor[]>
  - on_success: <processor[]>
  - output_format: <string>
  - tag: <string>
  - target_field: <ident>
  - timezone: <string>

Configuration​

FieldRequiredDefaultDescription
fieldY-Field containing the date to parse
formatsN-Array of expected date formats. Supports: standard Go time layouts, ISO8601, UNIX, UNIX_MS
descriptionN-Explanatory note
ifN-Condition to run
ignore_failureNfalseContinue processing if parsing fails
localeN-Locale for date parsing (e.g., "en_US")
on_failureN-See Handling Failures
on_successN-See Handling Success
output_formatN2006-01-02T15:04:05.000Z07:00Format pattern for the output timestamp
tagN-Identifier
target_fieldN@timestampField to store the parsed date
timezoneNUTCTimezone for parsing (e.g., "Europe/Amsterdam")

Details​

The processor features an advanced timestamp parser that can automatically detect and parse a wide variety of date/time formats without requiring explicit format specification. It supports nearly all timestamp formats, including:

Years
YYYY, YY, yyyy, yy
Months
MMMM (full name), MMM (abbreviated), MM (zero-padded), M
Days
DD, D, dd, d
Hours
HH (24-hour), hh (12-hour), h
Minutes / Seconds
mm, m, ss, s
Milliseconds
S, SSS, SSSSSS, SSSSSSSSS
Timezones
Z, ZZ, ZZZ
AM / PM
A, a

Human-readable and machine timestamps, as well as Java time patterns and Go time layouts are supported. Numeric timestamps (i.e. Unix epoch in seconds/milliseconds) are handled.

Output formats and timezones are customizable. Also, templates can be used for dynamic timezone and locale selection.

note

By default, the parsed date is stored in the @timestamp field, but this can be customized using target_field.

warning

Date parsing will fail if none of the specified formats match the input. Set ignore_failure to true if errors should be tolerated.

Examples​

Standard Dates​

Parsing a date using the standard format...

{
  "initial_date": "25/04/2016 12:02:01"
}
date:
  - field: initial_date
  - target_field: timestamp
  - formats: ["02/01/2006 15:04:05"]
  - timezone: "Europe/Amsterdam"

converts it to the ISO format with timezone:

{
  "initial_date": "25/04/2016 12:02:01",
  "timestamp": "2016-04-25T12:02:01.000+02:00"
}

Automatic Detection​

Parsing various timestamps without explicit specs...

{
  "date1": "2023-12-29_15:04:05",
  "date2": "29/Dec/2023:15:04:05 +0000",
  "date3": "Fri Dec 29 15:04:05 2023",
  "date4": "1461578521",
  "date5": "1461578521000"
}
date:
  - field: date1
  - target_field: time1
date:
  - field: date2
  - target_field: time2
date:
  - field: date3
  - target_field: time3
date:
  - field: date4
  - target_field: time4
date:
  - field: date5
  - target_field: time5

detects and normalizes all dates automatically:

{
  "time1": "2023-12-29T15:04:05.000Z",
  "time2": "2023-12-29T15:04:05.000Z",
  "time3": "2023-12-29T15:04:05.000Z",
  "time4": "2016-04-25T12:02:01.000Z",
  "time5": "2016-04-25T12:02:01.000Z"
}

Nested Fields with ISO8601​

Processing dates in nested objects...

{
  "user": {
    "login_date": "2016-04-25T12:02:01.789Z"
  }
}
date:
  - field: user.login_date
  - target_field: timestamp
  - formats: ["2006-01-02T15:04:05.000Z07:00"]

maintains the correct timestamp:

{
  "user": {
    "login_date": "2016-04-25T12:02:01.789Z"
  },
  "timestamp": "2016-04-25T12:02:01.789Z"
}

Output Formats​

Specifying a custom format...

{
  "initial_date": "25/04/2016 12:02:01"
}
date:
  - field: initial_date
  - target_field: timestamp
  - formats: ["02/01/2006 15:04:05"]
  - output_format: "2006-01-02 15:04:05"

generates the specified output:

{
  "initial_date": "25/04/2016 12:02:01",
  "timestamp": "2016-04-25 12:02:01"
}

Dynamic Timezone and Locale​

Template fields for timezone and locale...

{
  "event_date": "2016-04-25 12:02:01",
  "tz": "Europe/London",
  "lang": "en_GB"
}
date:
  - field: event_date
  - target_field: timestamp
  - formats: ["2006-01-02 15:04:05"]
  - timezone: "{{{tz}}}"
  - locale: "{{{lang}}}"

can be used for parsing:

{
  "event_date": "2016-04-25 12:02:01",
  "tz": "Europe/London",
  "lang": "en_GB",
  "timestamp": "2016-04-25T12:02:01.000+01:00"
}