Troubleshooting

This guide provides solutions to common Director deployment and operational issues. Issues are organized by category with step-by-step resolution procedures.

Installation Issues

Director Service Fails to Start

Symptoms:

• Service does not start after installation
• Error messages during service startup
• Director not appearing in management console

Resolution Steps:

1. Check System Requirements

   • Verify supported operating system version
   • Ensure minimum hardware requirements are met
   • Confirm administrator/root privileges for installation

2. Permission Validation

   • Ensure installation user has sufficient privileges
   • Check file and directory permissions in installation path
   • Verify service account permissions if using dedicated account

3. Port Availability

   • Confirm required ports are not in use by other services
   • Check local firewall settings
   • Identify port conflicts using network tools

4. Review Service Logs

   • Check installation script output for error messages
   • Review system event logs for service startup failures
   • Examine Director logs for specific error details

Diagnostic Commands:

PowerShell

# Check service status
Get-Service vmetric-director

# Check port usage
Get-NetTCPConnection -LocalPort 443 -ErrorAction SilentlyContinue
netstat -an | Select-String ":443"

# View service events
Get-EventLog -LogName System -Source "Service Control Manager" -Newest 20 |
  Where-Object { $_.Message -like "*vmetric*" }

Bash

# Check service status
systemctl status vmetric-director

# Check port usage
netstat -tulpn | grep :443
ss -tulpn | grep :443

# View service logs
journalctl -u vmetric-director -n 50

Director Waiting for Its First Configuration

Symptoms:

• A newly installed Director starts cleanly and connects to the platform, but ingests nothing
• Log message: "No configuration on disk. Waiting for the platform to deliver it before building the pipeline, because the storage tier is latched from it."
• A Warning every two minutes: "Still waiting for the platform configuration after 2m0s. The pipeline is not started until it arrives."
• The health endpoint reports "startup_phase": "awaiting_config"

This is a legitimate state rather than a failure. A Director's processing mode — In-memory, Persistent, or Crash-resistant — is chosen when the Director is created, read from the configuration the platform delivers, and fixed for the life of the process. Everything the pipeline is built from follows it: where records are buffered, whether the queues are backed by memory or disk, and whether a failed delivery is retried or discarded. A Director booting for the first time with nothing on disk therefore waits for that configuration instead of guessing at the mode.

Only a first boot waits:

Director	On startup
Receives its configuration from the platform, and has none on disk yet	Waits for the platform to deliver one. There is no time limit.
Receives its configuration from the platform, and already has one on disk	Starts immediately. The processing mode cannot have changed, so what is on disk is correct.
Self-managed	Starts immediately on its local configuration.
Standalone, with no fleet configuration	Starts immediately on its local configuration.

The wait adds nothing to the restart of a Director that already has a configuration.

Resolution Steps:

1. Confirm the Director is waiting rather than stuck

   • The message queue is already running and the Director is already connected to the platform — only the pipeline is held
   • awaiting_config sits between starting_message_queue and initializing_pipeline, so a Director reporting it has passed every earlier step

2. Check the link to the platform, not the Director

   • A Director that stays in this state has not been sent a configuration, so its listeners, its disk and its local ports are not where the fault is
   • Work through Director Cannot Connect to Cloud Platform

3. Have the platform deliver a configuration

   • On a first boot the Director asks the platform once, and nothing asks again if that request or its reply is lost
   • Any change to the Director in the web interface pushes a configuration, which ends the wait

4. If a configuration arrived but could not be used

   • The log reads "Can not use the delivered configuration for <director>", naming either the parse error or that the delivered configuration carries no environment
   • This is reported once and the Director waits for the next delivery, so correct the Director in the web interface and save it again

Diagnostic Commands:

PowerShell

# Check how far startup has progressed
(Invoke-WebRequest -Uri "http://127.0.0.1:15551/health").Content

Bash

# Check how far startup has progressed
curl -s http://127.0.0.1:15551/health

The health endpoint is served on the loopback address only, at the port set by debug.remote_console.port, which defaults to 15551.

VMMQ Connection Not Initialized

Symptoms:

• Log message: "vmmq connection is not initialized"
• Director starts but cannot process data
• Internal communication failures

Resolution Steps:

1. Verify NATS JetStream Configuration

   • Check that NATS server is running and accessible
   • Verify JetStream is enabled in the configuration
   • Confirm storage paths exist and have correct permissions

2. Check Storage Directory

   • Verify <vm_root>/Director/storage/nats/ directory exists
   • Ensure adequate disk space for JetStream storage
   • Check directory permissions

3. Review Configuration File

   • Validate YAML syntax in vmetric.yml
   • Confirm VMMQ settings are correctly specified
   • Check for configuration file corruption

Connection Issues

Director Cannot Connect to Cloud Platform

Symptoms:

• Director status shows "Not connected" in the web interface
• Connection verification fails during setup
• Timeout errors in Director logs

Resolution Steps:

1. Identify Your Regional Endpoint

   Your Director connects to the same regional cloud platform where you created it. To identify your region, check the URL in your browser when logged into VirtualMetric:

   Region	Endpoint
   Europe	https://app.eu-west.cloud.virtualmetric.com
   US	https://app.us-east.cloud.virtualmetric.com
   Australia	https://app.aus-east.cloud.virtualmetric.com

   info

   If you need to switch regions, log out and select a different region.

2. Validate Network Connectivity

   • Ensure port 443 is open for outbound HTTPS connections from the Director host
   • Verify DNS resolution is working properly
   • Test connectivity to your regional endpoint using the diagnostic commands below

3. Check Firewall Configuration

   • Allow outbound HTTPS/WebSocket Secure (443) to the VirtualMetric Cloud platform. For Layer 7 (application-aware) firewalls, a single rule for *.cloud.virtualmetric.com is sufficient. For firewalls without Layer 7 support — or security teams that require stricter, explicit rules — allow your regional service endpoint instead:
     • Europe: api-service.eu-west.cloud.virtualmetric.com
     • US: api-service.us-east.cloud.virtualmetric.com
     • Australia: api-service.aus-east.cloud.virtualmetric.com
   • Ensure no SSL/TLS inspection is blocking certificate validation
   • Verify proxy configurations if applicable

4. Validate API Key

   • Confirm the installation script was copied and executed correctly
   • Check for extra spaces or line breaks if you manually edited the script
   • Regenerate the API key from the Director's Connection tab in the web interface if corruption is suspected

5. Review System Time

   • Ensure system clock is synchronized
   • Use NTP to maintain accurate time synchronization
   warning

   TLS certificate validation requires accurate system time. A clock skew of more than a few minutes can cause silent connection failures.

Diagnostic Commands:

Use your regional endpoint (identified above) in the following commands:

PowerShell

# Set your regional endpoint
$region = "eu-west"  # Change to: us-east or aus-east

# Test DNS resolution
Resolve-DnsName "app.$region.cloud.virtualmetric.com"

# Check outbound connectivity
Test-NetConnection -ComputerName "app.$region.cloud.virtualmetric.com" -Port 443

# Verify TLS connectivity
Invoke-WebRequest -Uri "https://app.$region.cloud.virtualmetric.com" -Method Head -UseBasicParsing

Bash

# Set your regional endpoint
REGION="eu-west"  # Change to: us-east or aus-east

# Test DNS resolution
nslookup "app.$REGION.cloud.virtualmetric.com"

# Check outbound connectivity
nc -zv "app.$REGION.cloud.virtualmetric.com" 443

# Verify TLS connectivity
curl -I "https://app.$REGION.cloud.virtualmetric.com"

Connection Drops Intermittently

Symptoms:

• Director alternates between Connected and Not Connected states
• Periodic timeout errors in logs
• Data processing interruptions

Resolution Steps:

1. Network Stability Check

   • Monitor network latency and packet loss to your regional endpoint
   • Verify network equipment stability
   • Check for bandwidth limitations or throttling

2. Resource Monitoring

   • Ensure adequate CPU and memory resources
   • Monitor disk I/O performance
   • Check for resource contention with other services

3. Proxy Configuration

   • Verify proxy server stability and configuration
   • Check proxy authentication credentials
   • Consider bypassing proxy for VirtualMetric endpoints

Data Processing Issues

Director Not Receiving Data

Symptoms:

• No data appearing in target destinations
• Zero throughput metrics in monitoring
• Source systems showing successful transmission

Resolution Steps:

1. Source Configuration Verification

   • Confirm correct Director IP address in source system configuration
   • Verify port numbers match between source and Director
   • Check protocol settings (TCP/UDP for syslog, HTTP/HTTPS for APIs)

2. Network Connectivity Testing

   • Test connection from source system to Director
   • Verify routing and firewall rules allow traffic
   • Use packet capture tools to confirm data transmission

3. Director Input Configuration

   • Review device configuration in Director settings
   • Verify enabled protocols and listening ports
   • Check for configuration syntax errors

4. Log Analysis

   • Examine Director logs for input processing errors
   • Check for data parsing or validation failures
   • Review error messages for specific issues

Diagnostic Commands:

PowerShell

# Test syslog connectivity (UDP)
$udpClient = New-Object System.Net.Sockets.UdpClient
$bytes = [Text.Encoding]::ASCII.GetBytes("<14>Test message from PowerShell")
$udpClient.Send($bytes, $bytes.Length, "<director_ip>", 514)
$udpClient.Close()

# Check listening ports
Get-NetTCPConnection -LocalPort 514 -ErrorAction SilentlyContinue
Get-NetUDPEndpoint -LocalPort 514 -ErrorAction SilentlyContinue

Bash

# Test syslog connectivity
logger -n <director_ip> -P 514 "Test message"

# Check listening ports
netstat -tulpn | grep <director_ip>

# Monitor network traffic
tcpdump -i any port 514

Router Queue Errors

Symptoms:

• Log message: "There is no available log file for [logtype] in the router queue"
• Data processing appears stalled
• Empty storage/logs directory

Resolution Steps:

1. Verify Storage Configuration

   • Check that <vm_root>/Director/storage/ directory exists
   • Ensure subdirectories (data, logs, queue, sender) are present
   • Verify write permissions on storage directories

2. Check Disk Space

   • Ensure adequate free disk space for queue operations
   • Monitor disk usage during high-volume periods
   • Configure log rotation if disk space is limited
   warning

   If disk space is exhausted, queued data may be lost. Monitor storage usage proactively in high-volume environments.

3. Review Route Configuration

   • Verify routes are properly configured and enabled
   • Check that devices and targets are correctly linked
   • Validate pipeline configuration syntax

Data Processing Errors

Symptoms:

• Partial data processing with error messages
• Data transformation failures
• Inconsistent output formatting

Resolution Steps:

1. Pipeline Configuration Review

   • Validate YAML syntax in processing pipelines
   • Check field mappings and transformation rules
   • Verify regular expressions and parsing patterns

2. Data Format Validation

   • Examine sample input data for format consistency
   • Check for unexpected characters or encoding issues
   • Verify timestamp formats match expected patterns

3. Resource Monitoring

   • Monitor CPU and memory usage during processing
   • Check for disk space availability
   • Ensure adequate processing capacity for data volume

Performance Issues

High Resource Usage

Symptoms:

• Excessive CPU or memory consumption
• System performance degradation
• Out of memory errors

Resolution Steps:

1. Configuration Tuning

   • Adjust buffer sizes and queue lengths
   • Optimize processing batch sizes
   • Configure appropriate timeout values

2. Pipeline Optimization

   • Identify resource-intensive transformation operations
   • Optimize data parsing and enrichment logic
   • Consider sampling for high-volume data streams

3. System Monitoring

   • Implement comprehensive resource monitoring
   • Set up alerts for resource threshold breaches
   • Plan capacity upgrades based on usage patterns

Slow Data Processing

Symptoms:

• High latency between data ingestion and output
• Growing backlog of unprocessed data
• Timeout errors in processing pipeline

Resolution Steps:

1. Resource Optimization

   • Increase CPU and memory allocation to Director
   • Monitor resource utilization patterns
   • Consider vertical scaling for improved performance

2. Pipeline Efficiency

   • Review processing pipeline for optimization opportunities
   • Simplify complex transformation rules where possible
   • Optimize regular expressions and parsing logic

3. Output Destination Performance

   • Check target system capacity and response times
   • Verify network connectivity to destination systems
   • Consider batch processing for improved throughput

Target Uploads Starving

Symptoms:

• A Warning in section SenderUploadStarvation: "Upload starvation: target <name> (<type>): flushes averaged <x>% of batch_size <n>, …"
• The health endpoint reports sender_upload_starvation as warning, while the top-level status stays ready
• The same Warning appears in the health endpoint's recent_errors

A target starves its uploads when it sends batches far smaller than it could while its work piles up. Left alone, this ends in a hard stall: the in-memory payload store's router stage quota fills, routing blocks, and ingestion stops, while the health endpoint still reads ready. The Director reports it before that point.

It is measured on the in-memory payload store only, meaning the In-memory processing mode or persistent_storage.payload.type: memory. That store has the router stage quota that fills, and it is the only one that reports how much it holds. With the NATS object store, the default for persistent storage, or with inline payloads, which a Kafka or Event Hubs pipeline bus requires, the component stays ok.

A target is reported when all of these hold over the last 10 minutes:

Condition	Threshold
Its flushes are small	At least 3 flushes, averaging under 5% of its batch_size
Its own queue grows	By at least 10 notifications (payloads waiting for it)
The payload store grows	By at least 1 MiB

A down target is never reported, and neither are the healthy targets beside it. A target that is down piles up work but flushes nothing. Its own queue is what ties the store's growth to a target, so a healthy target that sends small batches is not blamed for another target's outage. For a target that is down, see Target Delivery Errors.

Resolution Steps:

1. Set a flush interval on the target

   • Set interval on the reported target so it accumulates records and sends fuller batches. See Scheduling and Pool Behavior
   • If the target already has an interval, lengthen it. sentinel, azmon and sentineldatalake default to a 30-second flush and can still be reported

2. Wait for the report to clear

   • The Warning repeats every 10 minutes while the condition lasts
   • The report clears after 10 minutes without the condition, with the Information line Target "<name>" is no longer starving its uploads.
   • A target removed or renamed while reported drops off without a clear line
   • If a configuration change moves the Director off the in-memory store, each reported target gets Target "<name>" is no longer reported for upload starvation: the payload store is no longer the in-memory one, where it is measured. (Information), and the component returns to ok

In the health endpoint, each reported target is one entry in the component's message, separated by semicolons:

"sender_upload_starvation": {"status":"warning","message":"<name> (<type>): flushes averaged …; <next target> …"}

warning is advisory. It never changes the top-level status, which stays ready, and probes do not restart the Director for it.

Measured target types are those that close a batch at a record count:

• Storage: awss3 and the other S3-compatible storage targets, azblob, gcpstorage, file
• Staging: azdx, and the Databricks, Redshift and Snowflake targets
• sentinel, azmon, sentineldatalake
• elastic, amazonopensearch, elasticsecurity
• splunk, splunksecurity, crowdstrike
• clickhouse, bigquery, gcpchronicle, gcplogging, gcpsecops, ibmcloudlogs
• mqtt, nats, synadia, rabbitmq, amqp, amazonmq
• http, https, otlp, tcp, datadog, sumologic, rapid7, splunks2s

Targets that batch by bytes or send each record on its own are not measured, such as kafka, eventhubs, azservicebus, gcppubsub, amazonkinesis, amazonsqs, amazonsns, amazoncloudwatch, syslog and udp.

Diagnostic Commands:

PowerShell

# Check the sender_upload_starvation component
(Invoke-WebRequest -Uri "http://127.0.0.1:15551/health").Content

Bash

# Check the sender_upload_starvation component
curl -s http://127.0.0.1:15551/health

Log Analysis

Director Log Locations

Windows:

<vm_root>\Director\storage\logs\

Linux:

<vm_root>/Director/storage/logs/

Common Log Messages

Message	Meaning	Action
vmmq connection is not initialized	NATS JetStream not available	Check NATS configuration and storage. A VMMQ: line near it usually names the cause
Can not update device state	Device status update failed	Verify VMMQ connectivity
Health check failed	Device collector unhealthy	Service will auto-restart collector
Can not fetch messages	Message queue read error	Check NATS server health
Failed to start VMMQ server	Internal messaging startup failed	Review configuration and permissions. A VMMQ: line near it usually names the cause
Durable <durable> on stream <stream> can not be loaded: its saved position (<file>, <n> bytes, last written <time>) is not a consumer state file, which a hard stop of the host leaves behind. Removing it and creating the durable again, starting at …	Warning. The host stopped hard around <time>, and the Director is recovering the durable. See the note on durable recovery below	None. To learn why the host stopped, look in the Windows System event log for Event ID 41 (Kernel-Power) or 6008 near <time>
Durable <durable> on stream <stream> was created again and starts at stream sequence <N>.	Information. The durable was recovered	None
Can not remove the saved position of durable <durable> on stream <stream>. Details: <error>	Error. The durable's store directory could not be removed, for example because antivirus locks a file or permissions deny it. The durable stays down, and the recovery is retried	Check the permissions on <vm_root>/Director/storage/nats/ and the antivirus exclusions
VMMQ: Error adding consumer '<account> > <stream> > <durable>': error creating store for consumer: corrupt state file (10104)	Warning. Logged at startup. This is the cause behind the durable recovery lines above	None. The Director recovers the durable
VMMQ: Filestore [<stream>] Stream state detected prior state, could not locate msg block <N>	Warning. After a hard stop: the messages in a block that never reached the disk were lost	None
VMMQ: Filestore [<stream>] Stream state outdated, last block has additional entries, will rebuild or … Recovering stream state from index errored: prior state file	Information. After any unclean stop, the stream's index is rebuilt from its messages. Nothing is lost	None
VMMQ: <n> server log lines were dropped: they came faster than the log is written.	Warning. The message queue logged faster than the log file is written	None
Can not hand a payload of <N> <type> records on, and it is kept to be handed on at the next rotation; …	Error. The payload store or NATS refused a listener's payload. It is kept and retried at every rotation, and nothing written after it is handed on before it. Repeated every ten minutes while it lasts, with (<N> attempts over <duration>, <M> since this was last logged)	Check NATS and the payload store. The listener slows or refuses its source once its buffer fills. See Listener Memory Sizing
A payload of <N> <type> records that could not be handed on is handed on, after <N> attempts over <duration>.	Information. The failure cleared and the kept payload went through	None
A payload of <N> <type> records could not be handed on in <N> attempts over <duration>, and the stream is stopping: it is dropped, …	Error. The collector stopped while the payload was still refused. Its records are dropped unacknowledged	None. A source that acknowledges after delivery sends the records again
A payload of <N> <type> records can never be handed on. It is held until the collector stops, …	Error. A listener that acknowledges holds a payload the store will never accept. The device ingests nothing until it is restarted	Restart the device. With persistent_storage.payload.type: inline, raise persistent_storage.payload.max_size
A payload of <N> <type> records can never be handed on, and it is dropped. Error: <reason>	Error. A listener that acknowledges nothing dropped a payload the store will never accept. At most every ten minutes per device, with (<M> more dropped since this was last logged)	With persistent_storage.payload.type: inline, raise persistent_storage.payload.max_size
Upload starvation: target <name> (<type>): flushes averaged <x>% of batch_size <n>, …	Warning. The target sends batches far smaller than its batch_size while its queue and the in-memory payload store grow. Repeated every 10 minutes while it lasts	Set a flush interval (interval) on the target, or lengthen it. See Target Uploads Starving
Target "<name>" is no longer starving its uploads.	Information. The upload starvation report cleared after 10 minutes without the condition	None

note

The Director service includes automatic recovery for transient failures. Messages like "Health check failed" are often followed by automatic collector restart without manual intervention.

Durable recovery after a hard stop

A hard stop of a Windows host can leave the saved position of a durable consumer on the Director's message queue unreadable. Power loss, a forced VM reset and a bugcheck do this; a service restart or a killed process does not. The message queue then refuses that durable, and the Director recovers it: it removes the unreadable position, creates the durable again, and starts it where it had stopped, so old requests are not acted on again and stats and agent data are not forwarded twice. Automation requests and fleet replies start at the first message since the message queue started, stats forwarding and agent data at the first message since the durable's position was last saved, and the ingester, router and sender queues at the first message still queued.

This applies only to a standalone Director's embedded message queue. A clustered Director's durables are kept by raft and are not touched, and a durable that an agent creates on the Director's message queue is not recovered.

The payload lines above carry the source FlushQueue. The upload starvation lines carry the section SenderUploadStarvation and the target's name. The durable recovery lines carry the section RecoverConsumer. Lines that start with VMMQ: come from the embedded message queue itself and carry the section VMMQServerLog. Its warnings and errors are always logged, and also appear in the health endpoint's recent_errors. A line that repeats is logged at most once every 30 seconds, followed by (<N> more like it since this was last logged). The message queue's notices are logged only with vmmq.debug.status: true. The message queue indents some of its lines, so in the log Error adding consumer follows VMMQ: after five spaces rather than one. Search for the text after the prefix. Failed to publish file notification for <locator> (source PublishFileNotification) is no longer logged: the rotation's line carries the error instead.

Target Delivery Errors

Every target sends through the same sender pool, so five message shapes wrap the target-specific reason. In the JSON log these entries carry "Section":"SenderPool" and the target name. The part after Reason:, or after the last colon, is the actual cause. Each target page explains its own causes in its Troubleshooting section.

Message	When	What happens to the data
ValidateConfig failed for target "<name>": <reason>	The configuration was rejected	Nothing is sent. The check repeats about every 30 seconds until the configuration changes.
Failed to reinitialize target "<name>" (attempt N). Reason: <reason>	Startup or reconnect failed	Retried after 5, 10, 20 and 40 seconds, then every 60 seconds. Only the first failure, and a failure whose reason changed, is logged as an Error. Repeats are Debug, and a Warning summary with the count of failed attempts follows at most every 15 minutes. See Reinitialize Failure Lines. Data waits in the queue and comes back about a second after the next attempt. The web interface shows the target as connection failed for <name>: <reason>.
Sender worker N ThreadSafeInit on reinit failed for "<name>": <reason>	A worker could not initialize	The worker retries. Targets that connect lazily often report a bad credential here for the first time while the connection status still shows connected.
Sender worker N execute() failed for <payload>: <reason>	Delivery failed in immediate mode	The payload is redelivered about every 5 seconds until it succeeds or the target classifies it as rejected. This holds while the target can initialize. A target that cannot initialize follows the reinitialize timing above. When the endpoint rejects the batch, it is dropped after 4 deliveries with deterministic failure for <payload> after 4 attempts — dropping (giving up). When the target refuses a single record before sending it, that record alone is dropped on the fourth delivery with Target "<name>" rejected record <n> of <payload> on its last delivery attempt — dropping that record and delivering the rest of the payload: <reason>, and the rest of the payload is delivered.
Sender worker N Finalize failed on flush for target "<name>": <reason>	A scheduled flush failed	The batch in memory is discarded, pending payloads are redelivered, and the target reinitializes. Rejected records are dropped after 4 deliveries with dropping <payload> for target "<name>" after 4 rejected flush attempts.

Two rules follow from this table:

• Retrying until fixed is the default. Only what the target classifies as rejected is ever dropped, after 4 deliveries: a batch the endpoint refuses with a permanent status is dropped whole, and a single record the target refuses before sending it is dropped alone. Any other refusal, even a deterministic one, for example a record that is too large or does not match the schema but is not classified as rejected, is retried indefinitely, and in immediate mode each retry reinitializes the target.
• A connected status does not prove the credentials work. Several targets make no network call at startup, so the first flush is the first real test of a credential or a permission.

Queued data is kept for queue.limit seconds (48 hours by default), and with the In-memory storage tier it does not survive a Director restart. See Persistent Storage.

Reinitialize Failure Lines

A target that cannot initialize logs its failed attempts per failure episode, which runs from the first failure until the target initializes again. Only the first failure and any change of reason are logged as an Error, so a single target that stays down does not fill the health endpoint's recent_errors on its own.

Line	Level	When
Failed to reinitialize target "<name>" (attempt N). Reason: <reason>	Error	The first failure of the episode
Failed to reinitialize target "<name>" (attempt N, reason changed). Reason: <reason>	Error	A failure whose reason differs from the previous one
Failed to reinitialize target "<name>" (attempt N). Reason: <reason>	Debug	Every repeat of the same reason
Target "<name>" still failing to reinitialize: <count> failed attempts since <time> (last at <time>). Reason: <reason>	Warning	At most once every 15 minutes while the same reason lasts
Target "<name>" reinitialized successfully (generation G) after <count> failed attempts since <time>.	Information	On recovery. One failure reads after 1 failed attempt, and the clause is left out when no attempt failed

Times are RFC 3339 in UTC.

• The attempt count is in the Warning and the recovery line. The Error shows only the first attempt, or the attempt at which the reason changed.
• Repeats appear only at debug.level 4 or higher, or in a live capture session. They are not kept in recent_errors.
• A reason that changes on every attempt stays at Error. Reasons are compared as whole text, so a reason carrying a changing part, such as a port, a request or statement ID, or a timestamp, logs every attempt as an Error tagged reason changed.
• Log-severity statistics count one Error per episode or reason change, plus a Warning at most every 15 minutes, rather than one Error per attempt. This matters for an alert rule that counts errors.

While the target is down, data queued for it is sent back to wait. Each message is due again about 1 second after the target's next reinitialize attempt can complete, never sooner than 5 seconds and never later than 2 minutes. At the 60-second retry interval, queued data comes back about once per attempt, and within about a second of the attempt that succeeds.

Log Analysis Commands

PowerShell

# Monitor real-time logs
Get-Content "<vm_root>\Director\storage\logs\director.log" -Wait -Tail 50

# Search for errors
Select-String -Path "<vm_root>\Director\storage\logs\*.log" -Pattern "error|failed" -AllMatches

Bash

# Monitor real-time logs
tail -f <vm_root>/Director/storage/logs/director.log

# Search for errors
grep -i "error\|failed" <vm_root>/Director/storage/logs/*.log

Support and Escalation

Clustered Deployments

For Directors in a cluster, troubleshoot each Director individually using the procedures above. Cluster health is an aggregation of individual Director states.

When standard troubleshooting procedures don't resolve issues:

1. Gather Diagnostic Information

   • Collect relevant log files and error messages
   • Document system configuration and environment details
   • Note specific symptoms and reproduction steps

2. Contact Support

   • Include Director version (vmetric-director -version)
   • Provide timeline of when issues first occurred
   tip

   Submit support tickets at support.virtualmetric.com. Include log excerpts and configuration snippets to expedite resolution.

3. Emergency Escalation

   • For critical production issues, use emergency contact procedures
   • Follow your organization's incident management processes