Targets: Overview

Director uses targets as the output destinations to forward, store, and analyze collected telemetry data. Targets provide flexible options for data persistence and integration with various analysis platforms.

Definitions

Targets serve as the endpoint destinations for processed data within the Director system. They operate on the following principles:

1. Output Configuration: Targets define where and how data should be delivered after processing.
2. Format Adaptation: They handle the conversion of internal data structures to destination-specific formats.
3. Delivery Management: Targets manage connection pooling, batching, and retry mechanisms.
4. Destination Integration: They provide authentication and protocol-specific features for various systems.

note

Targets enable:

Persistence: Local and remote storage with various retention options.
Integration: Seamless connection to analytics platforms and messaging systems.

They also support data transformation, and delivery confirmation.

Configuration

All targets share the following base configuration fields:

Field	Required	Default	Description
name	Y		Unique identifier for the target
description	N		Optional explanation
type	Y		Target type
status	N	true	Enable/disable the target
batch_size	N	1000	Records per batch

tip

Each target type provides specific configuration options detailed in their respective sections.

Use the name of the target to refer to it in your configurations.

Example:

targets:
  - name: elasticsearch
    type: elasticsearch
    properties:
      hosts: ["http://elasticsearch:9200"]
      index: "logs-%{+yyyy.MM.dd}"
      username: "elastic"
      password: "${PASSWORD}"

The target listed here is of type elasticsearch. It specifies the host that it will forward the data to and the index on the host to which the data will be appended. To access the host, it specifies a username and a password.

tip

You can use environment variables like ${PASSWORD} for your credentials. This will improve security by removing the credentials from your configuration file.

Deployment

The following deployment types can be used:

• One-to-many - data from a single source is routed to one or more destinations:

  Syslog → Local Storage + Analysis Platform

• Many-to-one - data from multiple sources is routed to one destination:

  Syslog + Windows → Local Storage

• Many-to-many - data from multiple sources is routed to multiple destinations:

  Syslog + Windows → Local Storage

  Syslog + Elasticsearch → Cloud Upload + Analysis Platform

• Chained - data is routed sequentially from one destination to the next:

  Syslog → Local Storage → Analysis Platform

Multiple targets can be used for redundancy, normalization rules can be implemented, and alerts can be put in place for notification and error handling.

Target Types

Targets can be categorized into several functional types that serve different data management needs:

• Analytical - These targets integrate with platforms designed for searching, analyzing, and visualizing data:

  • Elasticsearch, Splunk, and other search and analytics engines

• Storage-Based - These targets focus on data persistence with various retention strategies:

  • Local files, S3 and other cloud storage services

• Messaging - These targets publish data to distributed messaging systems:

  • Kafka, RabbitMQ, and other message brokers

• Integration - These targets connect with external systems through APIs:

  • Webhooks, REST endpoints, and custom integrations

Use Cases

The most common uses of targets are:

• Local analysis - Debug logging, performance analysis, audit trails, and temporary storage.

• Cloud integration - Long-term storage, data warehousing, security analysis, and compliance monitoring.

• Real-time analysis - Live monitoring, alert generation, trend analysis, and performance tracking.

• Data lake building - Raw data storage, schema evolution, data partitioning, and analytics preparation.

To serve these ends, the following processing options are available:

• Pipelines - Field normalization (for ECS, CIM, ASIM, CEF, LEEF, and CSL), data transformation, message batching, custom field mapping, schema validation, and format conversion.

• Buffer management - Configurable buffer sizes, batch processing, flush intervals, queue management, checkpoint recovery, and error handling.

• Performance - Asynchronous writing, buffer optimization, connection pooling, retry mechanisms, resource monitoring, and size-based rotation.

• Security - Authentication using API keys, service principals, and client certificates. Encryption with TLS/SSL, HTTPS, or custom algorithms. Also, access control and audit logging.

Implementation Strategies

Storage Types

Local

Director supports the following local data output methods:

• Console - Direct stdout writing with real-time message viewing. Also provides debugging and testing capability, format normalization, and synchronous writing with mutex locking

• Files - Multiple file formats like JSON (line-delimited), Avro (binary serialization), OCF (container), Parquet (columnar) are available. In addition, compression options like ZSTD (default), GZIP, Snappy, Brotli, and LZ4 are also supported. Also, dynamic file naming, size-based rotation, buffer management, and schema validation are among the features.

Cloud

The integration options for the cloud are below:

• Azure Blob - Direct blob writing and multiple containers are supported. Available authentication methods are service principal and managed Identity. Other features include automatic retries, exponential backoff, size-based chunking, connection pooling, and buffer management.

• Microsoft Sentinel - Direct DCR integration and ASIM normalization are supported. In addition to standard tables, WindowsEvent, SecurityEvent, CommonSecurityLog, and Syslog can be used. Various ASIM tables are also available. (See the ASIM section for a complete list.)

Redundant Delivery

For critical data flows, implement redundant delivery patterns:

• Primary and backup targets for important data
• Multi-region storage for disaster recovery
• Disk buffer with forward-on-recovery capability

Routing Optimization

Configure intelligent routing based on data characteristics:

• Route high-volume data to scalable storage targets
• Send critical alerts to real-time notification endpoints
• Direct compliance data to specialized archival systems