Version: 1.4.0

Routes: Overview

Routes define how incoming data is filtered, processed, and directed to specific destinations. They act as traffic controllers, determining what data goes through which pipelines and ultimately reaches which targets.

Definitions

Routes are made up of filtering expressions and sequences of processors that shape the format and flow of data streams. They curate data through an evaluation process and forward it to downstream consumers.

Their behavior is characterized by three key elements:

Devices - One or more sources to ingest data from
Pipelines - Optional processing schemes applied to selected events
Targets - One or more destinations to forward the processed data

In addition to these, there is one more consideration that involves the binary relations between devices and pipelines, pipelines and targets, or directly devices and targets: the optional filter expressions that match specific events.

Configuration

All routes share the following base configuration fields:

Field	Required	Default	Description
`name`	Y		Unique identifier for the route
`description`	N		Optional explanation of the route's purpose
`devices`	N		List of source IDs to collect events from
`if`	N		Expression determining if the route applies to an event
`pipelines`	N		Ordered list of pipelines to process matching events
`targets`	N		List of targets to deliver processed events to
`status`	N	`true`	Enable/disable the route

warning

Do not confuse field names like pipelines or targets with the component entries at the first level. As the highest level of a telemetry configuration, routes orchestrate the stream by specifying the components that constitute the traffic.

note

Enlisted devices, targets, and pipelines must be configured in advance to be usable by routes. Use their ids or names to refer to them.

Example:

routes:
  - name: syslog_to_backup
    devices: 
      - name: syslog_*
    pipelines: 
      - name: normalize
      - name: enrich
    targets: 
      - name: backup

This route receives data from devices with the prefix syslog_ in their names, transforms it using the normalize and enrich pipelines, and forwards it to the backup target.

Deployment

Forwarding data involves several design patterns (use of pipelines is optional in each case):

Single source to single target

Syslog → Route → [Normalize] → Storage
Single source to multiple targets

Syslog → Route → [Enrich] → Monitoring + Storage + Backup
Multiple sources to single target

Syslog + Apache → Route [Normalize] → Central Analytics
Multiple sources to multiple targets

Syslog + Winlog + Apache → Route [Normalize ∘ Enrich] → Cloud Upload + Central Analytics

Routes can include zero or more pipelines. Multiple pipelines will process data sequentially. A pipeline's processors can override routing using the reroute processor, and the final processor terminates execution.

Route Types

Routes can be categorized based on their function in the data flow:

Primary Routes - These routes handle critical data paths with high priority:
- Specific filtering, targeted processing, and specialized destinations
Filtering Routes - These routes select specific subsets of data:
- Data segregation based on categories, security levels, or compliance requirements
Processing Routes - These routes apply transformations to data:
- Apply enrichment, normalization, or specialized processing
Distribution Routes - These routes focus on delivering data to targets:
- Fan-out delivery, redundant storage, or multi-system integration
Fallback Routes - These routes catch any unmatched events:
- Default processing for events that don't match specific criteria

Use Cases

Some scenarios demonstrating the uses of routes:

Security Event Workflow:

routes:
  - name: security_critical
    if: "event.category == 'security' && event.severity == 'critical'"
    pipelines:
      - name: security_enrichment
      - name: threat_intelligence
    targets:
      - name: siem_platform
      - name: security_team_notification
  
  - name: security_standard
    if: "event.category == 'security'"
    pipelines:
      - name: security_enrichment
    targets:
      - name: security_elasticsearch

Data Segregation:

routes:
  - name: pci_data
    if: "tags.contains('pci')"
    pipelines:
      - name: pci_compliance
      - name: data_masking
    targets:
      - name: pci_storage
  
  - name: hipaa_data
    if: "tags.contains('hipaa')"
    pipelines:
      - name: hipaa_compliance
      - name: phi_protection
    targets:
      - name: hipaa_storage

Environment Separation:

routes:
  - name: production_logs
    if: "kubernetes.namespace == 'production'"
    pipelines:
      - name: production_enrichment
    targets:
      - name: production_elasticsearch
  
  - name: development_logs
    if: "kubernetes.namespace == 'development'"
    pipelines:
      - name: dev_enrichment
    targets:
      - name: development_elasticsearch

Implementation Strategies

The following aspects of your telemetry stream must be considered im the implementation of routes:

Conditional Routing:

Selecting events involves evaluating boolean expressions such as:

device.type == 'syslog'
dataset.name == 'firewall_logs'
log.severity > 4
!(source.ip == '10.0.0.1'||source.ip == '10.0.0.2')

Key points about route evaluation:
- Route conditions are evaluated before pipelines begin their work
- The first route that matches processes the event
- Pipelines can modify the flow or direction of the data
- Multiple targets receive identical copies of the same data
Examples of conditional routing:
```
routes:
  - name: critical_alerts
    if: "event.severity == 'critical'||event.severity == 'high'"
    pipelines:
      - name: alert_enrichment
    targets:
      - name: alerts_webhook
      - name: security_team_email
  
  - name: normal_logs
    if: "event.severity == 'low'||event.severity == 'medium'"
    pipelines:
      - name: standard_processing
    targets:
      - name: logs_elasticsearch
```

Priority-Based Processing:

Routes with higher priority—which is indicated by their relative position in the tree—are processed first:

routes:
  - name: urgent_alerts
    if: "event.severity == 'critical'"
    targets:
      - name: alerting_system
  
  - name: security_events
    if: "event.category == 'security'"
    targets:
      - name: security_platform
  
  - name: general_logs
    targets:
      - name: log_storage

Device Selection:

Routes can collect data from specific devices:

routes:
  - name: network_traffic
    devices:
      - name: firewall_logs
      - name: router_logs
      - name: switch_logs
    pipelines:
      - name: network_enrichment
    targets:
      - name: network_monitoring

Pipeline Chaining:

Routes can apply multiple pipelines in sequence:

routes:
  - name: web_security
    devices:
      - web_server_logs
    if: "event.category == 'web' && event.type == 'attack'"
    pipelines:
      - common_enrichment  # Applied first
      - attack_detection   # Applied second
      - threat_scoring     # Applied third
    targets:
      - security_elasticsearch

Multi-Target Delivery:

Routes can send processed data to multiple destinations:

routes:
  - name: compliance_events
    if: "tags.contains('compliance')"
    pipelines:
      - name: compliance_formatting
    targets:
      - name: compliance_elasticsearch  # For searching and analytics
      - name: compliance_s3             # For long-term retention
      - name: auditor_webhook           # For real-time notification

Fallback Routes:

Routes without conditions act as catch-all handlers for unmatched events:

routes:
  - name: primary_security
    if: "event.category == 'security'"
    targets:
      - name: security_elasticsearch
  
  - name: fallback_route
    # No if means this captures everything not matched above
    targets:
      - name: general_elasticsearch

Route Templates:

Use environment variables for flexible configurations.

routes:
  - name: dynamic_routing
    if: "${ROUTING_CONDITION}"
    pipelines:
      - ${PRIMARY_PIPELINE}
      - ${SECONDARY_PIPELINE}
    targets:
      - ${TARGET_SYSTEM}

Conditional Targets:

Apply different targets based on event properties.

routes:
  - name: adaptive_routing
    pipelines:
      - name: event_classifier
        processors:
          - set:
              field: _route_target
              value: "{{ event.priority > 7 ? 'high_priority' : 'standard' }}"
    targets:      
      - script:
        source:|
          if (ctx._route_target == 'high_priority') {
            return ['urgent_elasticsearch', 'notification_system'];
          } else {
            return ['standard_elasticsearch'];
          }

Definitions​

Configuration​

Deployment​

Route Types​

Use Cases​

Implementation Strategies​