Troubleshooting
This guide provides solutions to common Director deployment and operational issues. Issues are organized by category with step-by-step resolution procedures.
Connection Issues
Director Cannot Connect to Cloud Platform
Symptoms:
- Director status shows as "Not Connected" (red indicator)
- Connection verification fails during setup
- Timeout errors in Director logs
Resolution Steps:
-
Validate Network Connectivity
- Ensure port 443 is open for outbound HTTPS connections
- Verify DNS resolution is working properly
- Test connectivity:
curl -I https://portal.virtualmetric.com
-
Check Firewall Configuration
- Allow outbound connections to
*.virtualmetric.com
domains - Ensure no SSL/TLS inspection is blocking certificate validation
- Verify proxy configurations if applicable
- Allow outbound connections to
-
Validate API Key
- Confirm the API key was copied correctly during installation
- Check for extra spaces or line breaks in the key
- Regenerate API key if corruption is suspected
-
Review System Time
- Ensure system clock is synchronized (certificate validation requirement)
- Use NTP to maintain accurate time synchronization
- Check timezone configuration matches your location
Additional Diagnostics:
# Test DNS resolution
nslookup portal.virtualmetric.com
# Check outbound connectivity
telnet portal.virtualmetric.com 443
# Verify certificate chain
openssl s_client -connect portal.virtualmetric.com:443
Connection Drops Intermittently
Symptoms:
- Director alternates between Connected and Not Connected states
- Periodic timeout errors in logs
- Data processing interruptions
Resolution Steps:
-
Network Stability Check
- Monitor network latency and packet loss
- Verify network equipment stability
- Check for bandwidth limitations or throttling
-
Resource Monitoring
- Ensure adequate CPU and memory resources
- Monitor disk I/O performance
- Check for resource contention with other services
-
Proxy Configuration
- Verify proxy server stability and configuration
- Check proxy authentication credentials
- Consider bypassing proxy for testing
Installation Issues
Director Fails to Start
Symptoms:
- Installation script completes but service doesn't start
- Error messages during service startup
- Director not appearing in management console
Resolution Steps:
-
Check System Requirements
- Verify supported operating system version
- Ensure minimum hardware requirements are met
- Confirm administrator/root privileges for installation
-
Permission Validation
- Ensure installation user has sufficient privileges
- Check file and directory permissions in installation path
- Verify service account permissions if using dedicated account
-
Port Availability
- Confirm required ports are not in use by other services
- Check local firewall settings
- Use
netstat -an
to identify port conflicts
-
Review Installation Logs
- Check installation script output for error messages
- Review system event logs for service startup failures
- Examine Director service logs for specific error details
Common Solutions:
# Windows: Check service status
Get-Service VirtualMetricDirector
# Linux: Check service status
systemctl status virtualmetric-director
# Check port usage
netstat -tulpn | grep :443
Installation Script Fails
Symptoms:
- Script execution terminates with error
- Permission denied messages
- Download failures during script execution
Resolution Steps:
-
Execution Context
- Run PowerShell as Administrator (Windows)
- Use sudo privileges for bash script (Linux)
- Ensure execution policy allows script running (Windows)
-
Network Access
- Verify internet connectivity for downloading components
- Check corporate firewall for download restrictions
- Consider manual download if automated download fails
-
System Dependencies
- Install required runtime dependencies (.NET Core, Docker)
- Update package managers (apt, yum, chocolatey)
- Resolve any missing system libraries
Windows PowerShell Execution Policy:
# Check current execution policy
Get-ExecutionPolicy
# Allow script execution (if required)
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
Data Processing Issues
Director Not Receiving Data
Symptoms:
- No data appearing in target destinations
- Zero throughput metrics in monitoring
- Source systems showing successful transmission
Resolution Steps:
-
Source Configuration Verification
- Confirm correct Director IP address in source system configuration
- Verify port numbers match between source and Director
- Check protocol settings (TCP/UDP for syslog, HTTP/HTTPS for APIs)
-
Network Connectivity Testing
- Test connection from source system to Director
- Verify routing and firewall rules allow traffic
- Use packet capture tools to confirm data transmission
-
Director Input Configuration
- Review input configuration in Director settings
- Verify enabled protocols and listening ports
- Check for configuration syntax errors
-
Log Analysis
- Examine Director logs for input processing errors
- Check for data parsing or validation failures
- Review error messages for specific issues
Diagnostic Commands:
# Test syslog connectivity
logger -n <director_ip> -P <port> "Test message"
# Check listening ports
netstat -tulpn | grep <director_ip>
# Monitor network traffic
tcpdump -i any port <port_number>
Data Processing Errors
Symptoms:
- Partial data processing with error messages
- Data transformation failures
- Inconsistent output formatting
Resolution Steps:
-
Pipeline Configuration Review
- Validate YAML syntax in processing pipelines
- Check field mappings and transformation rules
- Verify regular expressions and parsing patterns
-
Data Format Validation
- Examine sample input data for format consistency
- Check for unexpected characters or encoding issues
- Verify timestamp formats match expected patterns
-
Resource Monitoring
- Monitor CPU and memory usage during processing
- Check for disk space availability
- Ensure adequate processing capacity for data volume
-
Error Log Analysis
- Review detailed error messages in Director logs
- Identify specific records causing processing failures
- Check for schema validation errors
Performance Issues
Slow Data Processing
Symptoms:
- High latency between data ingestion and output
- Growing backlog of unprocessed data
- Timeout errors in processing pipeline
Resolution Steps:
-
Resource Optimization
- Increase CPU and memory allocation to Director
- Monitor resource utilization patterns
- Consider vertical scaling for improved performance
-
Pipeline Efficiency
- Review processing pipeline for optimization opportunities
- Simplify complex transformation rules where possible
- Optimize regular expressions and parsing logic
-
Output Destination Performance
- Check target system capacity and response times
- Verify network connectivity to destination systems
- Consider batch processing for improved throughput
-
Clustering Consideration
- Evaluate clustered deployment for horizontal scaling
- Distribute processing load across multiple Director instances
- Implement load balancing for improved performance
High Resource Usage
Symptoms:
- Excessive CPU or memory consumption
- System performance degradation
- Out of memory errors
Resolution Steps:
-
Configuration Tuning
- Adjust buffer sizes and queue lengths
- Optimize processing batch sizes
- Configure appropriate timeout values
-
Pipeline Optimization
- Identify resource-intensive transformation operations
- Optimize data parsing and enrichment logic
- Consider caching frequently accessed data
-
System Monitoring
- Implement comprehensive resource monitoring
- Set up alerts for resource threshold breaches
- Plan capacity upgrades based on usage patterns
Migration and Maintenance
How to Migrate Director to New Server
Scenario: Moving Director installation to new hardware or different server.
Migration Steps:
-
Prepare New Environment
- Install required dependencies on new server
- Ensure network connectivity and firewall configuration
- Verify system requirements and permissions
-
Export Configuration
- Document current Director configuration and settings
- Export processing pipelines and custom rules
- Note integration points with source and destination systems
-
Install Director on New Server
- Run installation script on new server
- Use existing organization credentials and API key
- Follow standard installation verification procedures
-
Update System Integrations
- Update source systems to point to new Director IP address
- Verify destination system connectivity from new location
- Test data flow end-to-end
-
Decommission Old Director
- Stop data processing on old Director
- Remove Director from management console
- Properly dispose of old server and credentials
Director Reinstallation Process
When to Reinstall:
- API key compromise or security incident
- Significant configuration corruption
- Major system updates or changes
Reinstallation Steps:
-
Initiate Reinstallation from Console
- Navigate to Director details in management interface
- Click "Re-install director" button
- Acknowledge API key revocation warning
-
Execute New Installation Script
- Download new installation script with updated API key
- Run installation script with administrative privileges
- Verify successful installation and connection
-
Restore Configuration
- Reconfigure data sources and destinations
- Restore custom processing pipelines
- Verify all integrations are working properly
Important Notes:
- Previous API key will be immediately revoked
- All existing connections will be terminated
- Configuration may need to be restored manually
Advanced Troubleshooting
Log Analysis and Diagnostics
Director Log Locations:
- Windows:
C:\ProgramData\VirtualMetric\Director\Logs\
- Linux:
/var/log/virtualmetric/director/
Key Log Files:
application.log
- General Director operation and errorsconnection.log
- Cloud platform connectivity eventsprocessing.log
- Data transformation and pipeline executionperformance.log
- Resource usage and performance metrics
Log Analysis Commands:
# Monitor real-time logs
tail -f /var/log/virtualmetric/director/application.log
# Search for specific errors
grep -i "error\|exception" /var/log/virtualmetric/director/*.log
# Check recent connection events
grep "connection" /var/log/virtualmetric/director/connection.log | tail -50
Performance Monitoring
System Metrics to Monitor:
- CPU utilization and load average
- Memory usage and available space
- Disk I/O performance and space utilization
- Network throughput and connection statistics
Director-Specific Metrics:
- Data ingestion rate (events per second)
- Processing latency and queue depth
- Error rates and failure patterns
- Connection stability and uptime
Support and Escalation
When standard troubleshooting procedures don't resolve issues:
-
Gather Diagnostic Information
- Collect relevant log files and error messages
- Document system configuration and environment details
- Note specific symptoms and reproduction steps
-
Contact Support
- Submit support ticket with comprehensive diagnostic data
- Include Director version and deployment information
- Provide timeline of when issues first occurred
-
Emergency Escalation
- For critical production issues, use emergency contact procedures
- Follow your organization's incident management processes
- Consider temporary workarounds while permanent solutions are developed