Azure Monitor Logs

Microsoft Azure Logging & Monitoring

Synopsis

Creates a target that ingests log messages into Azure Monitor Logs (Log Analytics) workspace tables using Data Collection Rules (DCRs). Supports automatic table discovery, field normalization, and Azure Managed Identity authentication for secure, credential-free access.

tip

For Director Proxy deployment with Managed Identity, see VirtualMetric Director Proxy.

Schema

- name: <string>
  description: <string>
  type: azmon
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    function_app: <string>
    function_token: <string>
    rule_id: <string>
    endpoint: <string>
    streams:
      - name: <string>
        rule_id: <string>
    stream: <string[]>
    buffer_size: <numeric>
    batch_size: <numeric>
    keep_phantom_fields: <boolean>
    drop_unknown_stream_events: <boolean>
    cache:
      timeout: <numeric>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Core Settings

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be azmon
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Authentication

Field	Required	Default	Description
tenant_id	N*	-	Azure tenant ID (required for direct authentication)
client_id	N*	-	Azure client ID (required for direct authentication)
client_secret	N*	-	Client secret (required for direct authentication)
function_app	N*	-	Director Proxy endpoint URL (required for proxy forwarding)
function_token	N*	-	Director Proxy authentication token (required with function_app)

* = Conditionally required. Use either direct authentication (tenant_id, client_id, client_secret) OR Director Proxy forwarding (function_app, function_token), OR omit all credentials to use Managed Identity.

Managed Identity Authentication

Azure targets support Managed Identity authentication for credential-free access when Director is deployed on Azure infrastructure.

How it works: When tenant_id, client_id, and client_secret are omitted from the configuration, the target automatically uses Azure's DefaultAzureCredential, which attempts authentication in the following order:

1. Environment variables (AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET)
2. Workload Identity (for Kubernetes deployments)
3. Managed Identity (system-assigned or user-assigned)
4. Azure CLI credentials
5. Azure PowerShell credentials

Deployment requirement: Director must run on Azure infrastructure that supports Managed Identity:

• Azure Virtual Machines
• Azure App Service
• Azure Container Instances
• Azure Kubernetes Service (AKS)
• Azure Functions

Required permissions: The Managed Identity must be granted the appropriate Azure RBAC roles documented in each target's IAM Permissions section.

note

Managed Identity eliminates credential management overhead and is the recommended authentication method for Azure-hosted Director deployments.

Table Configuration

Field	Required	Default	Description
endpoint	Y		DCE Address or DCE Resource ID
rule_id	N		Default Data Collection Rule (DCR) ID
streams	N	-	Detailed stream (table) configurations
stream	N	-	Legacy string array of stream names
buffer_size	N	1048576	Encoding buffer size in bytes (1MB), applied per destination stream
batch_size	N	1000	Maximum messages per batch, counted per destination stream. Must be greater than 0
keep_phantom_fields	N	false	Keep fields not defined in DCR schema
drop_unknown_stream_events	N	true	Silently drop events for undefined tables. With false, such an event is rejected with an error, retried 3 times, then dropped
cache.timeout	N	300	Table cache timeout in seconds. In autodiscovery mode, also how often the DCRs are read again. 0 or less turns that refresh off
field_format	N	-	Data normalization format. See applicable Normalization section

Scheduling

warning

Unlike most target types, azmon does not flush once per notification when left unconfigured. With neither interval nor cron set it flushes every 30 seconds.

See Scheduling and Pool Behavior for the interval, cron and max_items fields shared by all targets, and for how to override or opt out of this default.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The Azure Monitor Logs target enables direct ingestion into Log Analytics workspace tables using the modern Data Collection Rules (DCRs) API. This approach provides secure authentication with Entra ID, schema validation, and optional data transformation through DCRs.

Deployment Models

The target supports two deployment models:

Direct Authentication - Director connects directly to Azure using service principal credentials (tenant_id, client_id, client_secret). This model requires Director to have network connectivity to Azure endpoints and credentials for the target subscription.

Director Proxy Forwarding - Director sends processed data to VirtualMetric Director Proxy (Azure Function) deployed in customer environment. Director Proxy uses Azure Managed Identity for credential-free access to Azure Monitor Logs, eliminating the need to share Azure credentials with Director.

The Director Proxy model is particularly valuable for MSSP deployments where customers maintain complete control over Azure credentials while enabling centralized data processing and routing by the MSSP's Director infrastructure.

IAM Permissions

When using Service Principal or Managed Identity authentication, the following Azure roles are required for runtime data ingestion:

Azure Role	Scope	Purpose
Monitoring Metrics Publisher	Data Collection Rule	Send data via DCR ingestion API
Monitoring Reader	Resource Group	List DCEs and DCRs for stream auto-discovery

The Monitoring Reader role can also be assigned per resource instead of on the resource group. Autodiscovery works with either model, and the same scopes apply to the managed identity of a Director Proxy. Never assign it at subscription scope:

Azure Role	Scope	Purpose
Monitoring Reader	Data Collection Endpoint	Read the DCE and find the DCRs linked to it
Monitoring Reader	Each Data Collection Rule	Read the DCR's stream and table declarations
Monitoring Metrics Publisher	Each Data Collection Rule	Send data via DCR ingestion API

A DCR that is missing a role is left out of discovery and only its tables fail. See the Troubleshooting section of the Microsoft Sentinel target for the resulting errors.

If the setup functions (DCE/DCR/table creation) are used, additional roles are required:

Azure Role	Scope	Purpose
Monitoring Contributor	Resource Group	Create and update DCEs and DCRs
Log Analytics Contributor	Log Analytics Workspace	Create and update custom log tables

Dynamic Table Routing

The target supports using the SystemS3 field to route messages to specific tables. When a log message contains a SystemS3 field, it overrides the default table selection, using the format Custom-TableName.

Schema Validation

The target automatically detects table schemas from your Data Collection Rules and validates incoming data. When keep_phantom_fields is set to false, the target removes fields that aren't defined in the DCR schema before sending data.

warning

Disabling keep_phantom_fields may result in data loss for undefined fields.

Batch Processing

Data is buffered per destination stream until batch_size records or buffer_size bytes are reached for that stream, or the flush schedule fires. Whichever limit binds first triggers the upload; the record that would have exceeded it rides the next one rather than being dropped. Because both limits are per stream, a target fanning out to several streams accumulates up to that much for each of them before a flush. Each table type also has its own ceiling in the Log Analytics ingestion API.

warning

Large buffer sizes or batch sizes increase memory usage, and both are allocated per destination stream — a target writing to many streams multiplies the cost.

Field Normalization

The field_format property allows normalizing log data to standard formats before ingestion. Field normalization is applied before the logs are sent to Azure Monitor, ensuring consistent indexing and search capabilities.

Unknown Events Handling

When drop_unknown_stream_events is enabled, the target silently discards events destined for tables that don't exist in your DCR configuration. This prevents errors but may result in data loss.

When it is disabled, such an event is rejected instead, and logged as record rejected by target: no DCR stream for table "<table>" (stream "<stream>" is not configured on the target), or as no DCR stream for definition <N> for an event with no table. The target stays up and does not reinitialize. The payload is retried about every 2 seconds, and on its 4th delivery it is dropped (dropping (giving up)) and its undelivered events are added to the target's dropped counter. Events before the rejected one in the same batch are delivered, and events after it are dropped with it. So disabling it does not hold data until the table is configured: it makes the drop visible, then drops the data all the same.

warning

Enabling drop_unknown_stream_events silently discards unmatched events.

Autodiscovery

Director provides an autodiscovery feature that automatically configures Data Collection Rules and their associated tables.

The required permissions are:

• For Data Collection Rules

  Role	Scope
  Monitoring Metrics Publisher	Each DCR with name starting with vmetric

• For Resource Groups

  Role	Scope
  Monitoring Reader	Resource Group containing your DCE

As an alternative to the resource group, Monitoring Reader can be assigned on the DCE and on each DCR individually. Both models are described under IAM Permissions above.

warning

Assign the Monitoring Reader role at the Resource Group level or on the individual DCE and DCRs, never at the Subscription level.

The feature uses Resource IDs to discover DCRs and their configurations. It then automatically detects table schemas and validates fields, and prevents phantom fields through schema validation.

While the target runs, Director reads the DCRs again every cache.timeout seconds (default 300), in the background, and applies your own streams list to the result as it does at startup. Sends never wait for that read: each worker picks up the new set of tables at its next send.

• A table that a new or newly readable DCR adds is used from the next send.
• A table whose DCR, endpoint or columns changed, or that is no longer discovered, is switched or dropped only after the records already buffered for it have been delivered. A refresh never discards a buffered record.
• A refresh that fails, or that finds no tables, logs a Warning and keeps the tables from the last successful discovery.

The refresh is off when cache.timeout is 0 or less, or when endpoint is an https:// URL, which turns autodiscovery off.

Examples

Basic Configuration

Configuration using Resource ID-based autodiscovery:

targets:
  - name: basic_azmonitor
    type: azmon
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"

Managed Identity

Using Managed Identity Authentication instead of App Registration:

targets:
  - name: managed_identity_azmonitor
    type: azmon
    properties:
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"

Director Proxy

Configuration using Director Proxy for credential-free forwarding:

targets:
  - name: proxy_azmonitor
    type: azmon
    properties:
      function_app: "https://my-director-proxy.azurewebsites.net/api/Sentinel"
      function_token: "your-proxy-authentication-token"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"

Specific Tables

Using specific table filtering and custom cache timeout:

targets:
  - name: filtered_azmonitor
    type: azmon
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      streams:
        - name: "Custom-ApplicationLogs"
        - name: "Custom-PerformanceMetrics"
      cache:
        timeout: 300
      keep_phantom_fields: false
      drop_unknown_stream_events: true

Custom Tables

Configuration for custom log tables:

targets:
  - name: custom_tables_azmonitor
    type: azmon
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      streams:
        - name: "Custom-WebServerLogs"
          rule_id: "dcr-00000000000000000000000000000001"
        - name: "Custom-DatabaseLogs"
          rule_id: "dcr-00000000000000000000000000000002"
      keep_phantom_fields: true

High-Volume

Optimization for high-volume ingestion:

targets:
  - name: highvolume_azmonitor
    type: azmon
    pipelines:
      - normalization
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      batch_size: 5000
      streams:
        - name: "Custom-HighVolumeLogs"

Multiple Workspaces

Configuration for sending to different workspaces:

targets:
  - name: production_azmonitor
    type: azmon
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/production/providers/Microsoft.Insights/dataCollectionEndpoints/prod-dce"
      streams:
        - name: "Custom-ProductionLogs"

  - name: staging_azmonitor
    type: azmon
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/staging/providers/Microsoft.Insights/dataCollectionEndpoints/staging-dce"
      streams:
        - name: "Custom-StagingLogs"

Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_azmonitor
    type: azmon
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      field_format: "ecs"
      streams:
        - name: "Custom-NormalizedLogs"

Scheduled Batching

Configuration with scheduled batch delivery:

targets:
  - name: scheduled_azmonitor
    type: azmon
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      batch_size: 1000
      interval: "5m"

With Debugging

Configuration with debug options:

targets:
  - name: debug_azmonitor
    type: azmon
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      debug:
        status: true
        dont_send_logs: true

Querying Data

After ingestion, query your logs in Log Analytics using Kusto Query Language (KQL):

Custom_ApplicationLogs_CL
| where TimeGenerated > ago(1h)
| where severity_s == "error"
| summarize count() by host_s

note

Custom tables created via DCR have the _CL suffix automatically appended by Azure Monitor.

Troubleshooting

The azmon target authenticates, discovers streams, and uploads in exactly the same way as the sentinel target, so it reports the same errors for the same causes: missing Monitoring Reader or Monitoring Metrics Publisher roles, credential errors, an empty stream configuration, rejected uploads, and Director Proxy errors. Use the Troubleshooting section of the Microsoft Sentinel target for the full list of errors, causes, and fixes, and Target Delivery Errors for how Director logs and retries target failures.

Two differences apply when reading that section for this target:

• Log lines and the connection status carry this target's name, so match on the cause text, which is the part after Reason: or after the last colon, rather than on the target name shown in the examples.
• The target wizard in the web interface offers plain Director proxy inputs for the endpoint URL and the token, without a proxy picker, so a stored proxy entry cannot be selected there. The two values map to function_app and function_token, and the errors in the Director Proxy table apply unchanged.