Normalization

Normalization is a critical stage connecting ingestion from sources and forwarding to targets used to coalesce log data from diverse sources into consistent formats, enabling unified handling across different logging systems.

Log Formats

The processor supports several widely-used log formats:

Generic

Format	Notation	Key Identifier	Layout Characteristics	Example Fields
Elastic Common Schema (ECS)	Dot notation with lowercase	@timestamp	Hierarchical structure	source.ip, network.direction
Splunk Common Information Model (CIM)	Underscore with lowercase	_time	Flat structure	src and src_ip (both emitted), direction
Advanced Security Information Model (ASIM)	PascalCase	TimeGenerated	Explicit names	SrcIpAddr, NetworkDirection
Google SecOps Unified Data Model (UDM)	Nested structure	metadata.event_timestamp	Entity-based hierarchy	principal.ip, target.ip
Open Cybersecurity Schema Framework (OCSF)	Nested structure	time	Class-based hierarchy	src_endpoint.ip, dst_endpoint.ip
Graylog Extended Log Format (GELF)	Underscore prefix	version + host	Flat structure	_source_ip, _network_direction
OpenTelemetry (OTel)	Dot notation with lowercase	None — set explicitly	Flat structure with semantic-convention keys	source.address, severity_text

Security-specific

Format	Description	Key Identifier	Example Fields
Common Event Format (CEF)	ArcSight's standard format	rt (receiptTime)	src, dpt
Log Event Extended Format (LEEF)	IBM QRadar's format	devTime	src, srcPort
Common Security Log (CSL)	Microsoft Sentinel's format	TimeGenerated	SourceIP, DestinationPort

Format Detection

Source formats can be automatically detected using certain characteristic fields, e.g.

Context	Field	Format
Timestamp	@timestamp	ECS
	_time	CIM
	TimeGenerated	ASIM/CSL
	metadata.event_timestamp	UDM
	time	OCSF
Security	rt	CEF
	devTime	LEEF
CSL detection	TimeGenerated + LogSeverity	CSL
	TimeGenerated only	ASIM
UDM detection	metadata.event_timestamp	UDM
OCSF detection	class_uid	OCSF
GELF detection	version + host + short_message	GELF
OTel detection	None	OTel

OTel is not auto-detected

Unlike every other format, OTel has no single field that identifies it. To normalize a record that is already OTel-shaped, source_format: otel must be set explicitly. It is inferred automatically only when an earlier normalize step in the same pipeline already produced an otel record.

Conversion

Casing and Delimiters

Each format follows specific naming conventions:

ECS	source.ip, event.severity
CIM	src, severity
ASIM	SrcIpAddr, EventSeverity
CEF	src, Severity
LEEF	src, sev
CSL	SourceIP, LogSeverity
UDM	principal.ip, security_result.severity
OCSF	src_endpoint.ip, severity
GELF	_source_ip, level
OTel	source.address, severity_text

caution

Complex format conversions may impact performance.

Field Mapping

There are identifiable common network fields based on context across various formats:

	Context
Format	Source IP	Destination IP	Direction
ecs	source.ip	destination.ip	network.direction
cim	src	dest	direction
asim	SrcIpAddr	DstIpAddr	NetworkDirection
cef	src	dst	deviceDirection
leef	src	dst	deviceDirection
csl	SourceIP	DestinationIP	CommunicationDirection
udm	principal.ip	target.ip	network.direction
ocsf	src_endpoint.ip	dst_endpoint.ip	connection_info.direction
gelf	_source_ip	_destination_ip	_network_direction
otel	source.address	destination.address	network.direction

OTel has no IP-only field for an endpoint. source.address and destination.address receive the ECS source.ip and destination.ip, but under the semantic conventions the same fields can also hold a hostname or a socket path. Do not assume that a value there is always an IP address.

Configuration

The following fields are used to define the component:

Basic

Convert from ECS to ASIM format:

normalize:
  source_format: ecs
  target_format: asim

Field-specific

Convert a specific network field:

normalize:
  field: network_data
  source_format: cef
  target_format: ecs

Auto-detection

Let the processor detect the source format:

normalize:
  target_format: cim

UDM Conversion

Convert ECS to Google SecOps UDM format:

normalize:
  source_format: ecs
  target_format: udm

OCSF Conversion

Convert to Amazon Security Lake OCSF format:

normalize:
  source_format: ecs
  target_format: ocsf

Preprocessing

Fields are standardized with normalize for conversion between the ECS, CIM, ASIM, CEF, LEEF, CSL, OCSF, UDM, GELF, and OTel formats (see the Log Formats and Conversion sections above). Values are formatted for uniform casing with uppercase and lowercase processors when required by the target format's naming conventions.

Postprocessing

Fields are optimized for storage and queries using format conversion with the normalize processor (see the Conversion and Field Mapping sections above). For Microsoft Sentinel integration, data is prepared by converting to the ASIM format with normalize. For Google SecOps, convert to UDM format. For Amazon Security Lake, convert to OCSF format (see Log Formats table).

Schema Enforcement

Schema enforcement runs by default for six target formats — ASIM, CSL, OCSF, UDM, GELF and OTel. The rules normalize timestamps, validate event types, normalize security actions and bring field values into line with the target schema. ecs, cef and leef apply no enforcement pass.

warning

Complex format conversions may impact processing performance and delivery latency.