Skip to main content
Version: 1.4.0

Introduction

Directors are the core data processing engines within the DataStream platform, responsible for collecting, processing, transforming, and routing security telemetry data from various sources to target destinations. They serve as the central orchestration layer that maintains data sovereignty by keeping sensitive information within your environment while providing centralized cloud-based management.

What is a Director?​

A Director is a lightweight, containerized service that acts as a secure data processing hub in your infrastructure. It connects securely to the DataStream cloud platform for configuration management while ensuring all sensitive security data remains within your controlled environment.

Key Capabilities​

Data Processing Pipeline:

  • Ingests security data from multiple sources (syslog, APIs, files, databases)
  • Applies real-time transformation and normalization using YAML-defined pipelines
  • Supports multiple security schemas (ASIM, OCSF, ECS, CIM, UDM)
  • Routes processed data to various destinations (SIEM platforms, data lakes, security tools)

Security and Compliance:

  • Maintains data sovereignty by processing all data locally
  • Establishes outbound-only HTTPS connections to cloud management services
  • Provides comprehensive audit logging and activity tracking
  • Supports enterprise security requirements and compliance frameworks

Scalability and Reliability:

  • Horizontal scaling through clustering capabilities
  • High availability configurations for mission-critical environments
  • Resource-efficient processing with minimal infrastructure requirements
  • Automatic failover and load balancing in clustered deployments

Platform Management Options​

DataStream provides two distinct management approaches for Directors, each designed for different organizational needs and security requirements:

Self-Managed Directors​

Self-Managed Directors provide complete control over the deployment and management of your data processing infrastructure. This option is ideal for organizations with specific security requirements or existing infrastructure management processes.

Characteristics:

  • Full control over deployment environment and configuration
  • Direct management of updates, patches, and maintenance
  • Custom security controls and compliance configurations
  • Integration with existing infrastructure monitoring and management tools
  • Support for air-gapped or restricted network environments

Suitable For:

  • Organizations with strict data governance requirements
  • Environments with existing container orchestration systems
  • Companies requiring custom security configurations
  • Regulated industries with specific compliance needs

Managed Directors (Enterprise Feature)​

Managed Directors offer a fully-managed service where VirtualMetric handles the infrastructure management, monitoring, and maintenance of your Directors while still maintaining data sovereignty.

Characteristics:

  • Automated deployment and configuration management
  • Proactive monitoring and maintenance by VirtualMetric
  • Automatic updates and security patches
  • 24/7 support and incident response
  • Performance optimization and capacity planning

Suitable For:

  • Organizations seeking reduced operational overhead
  • Teams without dedicated infrastructure management resources
  • Companies prioritizing time-to-value over operational control
  • Environments requiring guaranteed SLA and support coverage

Installation Types​

Directors support different installation architectures to accommodate various operational requirements and scale needs:

Standalone Installation​

Standalone is the default installation type, designed for straightforward deployments where a single Director instance handles all data processing needs.

Features:

  • Single Director instance per deployment
  • Simplified configuration and management
  • Resource-efficient for most use cases
  • Quick deployment and setup process

Limitations:

  • No built-in high availability or load balancing
  • Single point of failure for data processing
  • Limited horizontal scaling capabilities
  • Manual backup and disaster recovery procedures

Recommended For:

  • Small to medium-scale deployments
  • Development and testing environments
  • Organizations with basic availability requirements
  • Initial proof-of-concept implementations

Clustered Installation (Enterprise Feature)​

Clustered installations provide high availability and horizontal scaling capabilities through multiple Director instances working together.

Features:

  • Multiple Director instances with automatic load balancing
  • Built-in failover and redundancy mechanisms
  • Horizontal scaling based on processing demands
  • Distributed processing for improved performance
  • Shared state management across cluster nodes

Benefits:

  • Elimination of single points of failure
  • Improved processing capacity and throughput
  • Automatic recovery from node failures
  • Dynamic scaling based on data volume
  • Enhanced monitoring and observability

Recommended For:

  • Mission-critical security data processing
  • High-volume environments requiring guaranteed availability
  • Organizations with strict SLA requirements
  • Production deployments requiring enterprise-grade reliability

Directors Management Interface​

The Directors interface provides comprehensive tools for monitoring and managing your Director fleet across different environments and deployment types.

Directors Dashboard​

The main Directors interface offers centralized visibility and control over all Director instances:

Directors Table:

  • Name - Unique identifier assigned during Director creation
  • Platform Type - Management model (Self-managed or Managed)
  • Installation Type - Architecture type (Standalone or Clustered)
  • Status - Current operational state (Enabled/Disabled)
  • Connection Status - Real-time connectivity indicator (Connected/Not Connected)

Management Controls:

  • Search directors - Quick filtering by Director name
  • Status filter - Filter by operational status
  • Create director - Initiate new Director deployment
  • Actions menu (⋮) - Per-Director operations (Edit, Enable/Disable, Delete)

Director Operations​

The platform provides comprehensive management capabilities for the complete Director lifecycle:

Creating Directors:

  • Guided setup process with name assignment and configuration selection
  • Platform-specific installation scripts (PowerShell, Bash)
  • Automatic API key generation for secure cloud connectivity
  • Connection verification and status monitoring

Managing Directors:

  • Real-time status monitoring and health checks
  • Configuration updates and deployment management
  • Activity logging and audit trail maintenance
  • Connection troubleshooting and diagnostic tools

Monitoring Directors:

  • Performance metrics and resource utilization tracking
  • Data throughput and processing statistics
  • Error logging and incident tracking
  • Integration status with source systems and destinations

Director Architecture and Data Flow​

Directors operate as secure intermediaries between your security data sources and target destinations, implementing a data sovereignty model that keeps sensitive information within your controlled environment.

Data Processing Architecture​

Input Layer:

  • Multiple simultaneous data source connections
  • Protocol-agnostic ingestion (Syslog, REST APIs, file monitoring)
  • Real-time streaming and batch processing capabilities
  • Built-in buffering and queuing for reliability

Processing Layer:

  • YAML-defined transformation pipelines
  • Multi-schema normalization and enrichment
  • Real-time data validation and quality checks
  • Custom logic implementation through processors

Output Layer:

  • Multi-destination routing and delivery
  • Format adaptation for different target systems
  • Delivery confirmation and retry mechanisms
  • Performance optimization for various endpoint types

Security and Connectivity Model​

Outbound-Only Communication:

  • Directors initiate all cloud platform connections
  • No inbound firewall rules required
  • Encrypted HTTPS communication for all cloud interactions
  • Certificate-based authentication and authorization

Data Sovereignty:

  • All security data processing occurs locally
  • No sensitive data transmitted to cloud services
  • Configuration and metadata-only cloud synchronization
  • Complete audit trail for compliance and governance

Getting Started with Directors​

To begin using Directors in your environment, follow this general workflow:

  1. Access Director Management - Navigate to Home > Fleet Management > Directors
  2. Create New Director - Use the guided setup to define your Director configuration
  3. Deploy Director Service - Run the provided installation script in your target environment
  4. Verify Connectivity - Confirm successful connection to the DataStream cloud platform
  5. Configure Data Sources - Set up connections to your security data sources
  6. Define Processing Rules - Create YAML pipelines for data transformation and routing
  7. Monitor Operations - Use the dashboard to track performance and troubleshoot issues

The Directors interface provides comprehensive guidance and support throughout this process, with detailed documentation, troubleshooting resources, and expert assistance available for complex deployments.