Skip to main content

Microsoft Sentinel data lake

Microsoft Azure Security Lake

Synopsis​

Creates a target that ingests log messages into Microsoft Sentinel data lake tables with lower ingestion costs and extended retention capabilities. Optimized for high-volume, high-fidelity log types like firewall logs, DNS logs, and network traffic requiring long-term storage.

tip

For more details on Microsoft Sentinel integration, refer to Microsoft Sentinel Overview and Microsoft Sentinel Integration. For Director Proxy deployment, see VirtualMetric Director Proxy.

Schema​

- name: <string>
  description: <string>
  type: sentineldatalake
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    function_app: <string>
    function_token: <string>
    rule_id: <string>
    endpoint: <string>
    streams:
      - name: <string>
        rule_id: <string>
    stream: <string[]>
    buffer_size: <numeric>
    batch_size: <numeric>
    keep_phantom_fields: <boolean>
    drop_unknown_stream_events: <boolean>
    cache:
      timeout: <numeric>
    field_format: <string>
    detection_rules:
      - name: <string>
        query: <string>
        language: <string>
    unmatched_tables: <enum>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration​

The following fields are used to define the target:

Core Settings​

FieldRequiredDefaultDescription
nameYTarget name
descriptionN-Optional description
typeYMust be sentineldatalake
pipelinesN-Optional post-processor pipelines
statusNtrueEnable/disable the target

Authentication​

FieldRequiredDefaultDescription
tenant_idN*-Azure tenant ID (required for direct authentication)
client_idN*-Azure client ID (required for direct authentication)
client_secretN*-Client secret (required for direct authentication)
function_appN*-Director Proxy endpoint URL (required for proxy forwarding)
function_tokenN*-Director Proxy authentication token (required with function_app)

* = Conditionally required. Use either direct authentication (tenant_id, client_id, client_secret) OR Director Proxy forwarding (function_app, function_token), OR omit all credentials to use Managed Identity.

Managed Identity Authentication​

Azure targets support Managed Identity authentication for credential-free access when Director is deployed on Azure infrastructure.

How it works: When tenant_id, client_id, and client_secret are omitted from the configuration, the target automatically uses Azure's DefaultAzureCredential, which attempts authentication in the following order:

  1. Environment variables (AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET)
  2. Workload Identity (for Kubernetes deployments)
  3. Managed Identity (system-assigned or user-assigned)
  4. Azure CLI credentials
  5. Azure PowerShell credentials

Deployment requirement: Director must run on Azure infrastructure that supports Managed Identity:

  • Azure Virtual Machines
  • Azure App Service
  • Azure Container Instances
  • Azure Kubernetes Service (AKS)
  • Azure Functions

Required permissions: The Managed Identity must be granted the appropriate Azure RBAC roles documented in each target's IAM Permissions section.

note

Managed Identity eliminates credential management overhead and is the recommended authentication method for Azure-hosted Director deployments.

Stream Configuration​

FieldRequiredDefaultDescription
endpointYDCE Address or DCE Resource ID
rule_idN-Default Data Collection Rule (DCR) ID
streamsN-Array of stream configurations with name and optional rule_id
streamN-Legacy string array of stream names
buffer_sizeN1048576Encoding buffer size in bytes (1MB), applied per destination stream
batch_sizeN1000Maximum messages per batch, counted per destination stream. Must be greater than 0
keep_phantom_fieldsNfalseKeep fields not defined in DCR schema
drop_unknown_stream_eventsNtrueSilently drop events for undefined streams. With false, such an event is rejected with an error, retried 3 times, then dropped
cache.timeoutN300Stream cache timeout in seconds. In autodiscovery mode, also how often the DCRs are read again. 0 or less turns that refresh off
field_formatN-Data normalization format. See applicable Normalization section

Detection Rules​

A detection rule set assigned to this target turns its smart pipeline on: the target compiles the set with the Smart Engine processor and ships only the events those detections could need, field- and value-reduced.

In the web interface these entries are written for you — assign rules from Content Management > Detections, or from this target's own Detection Rules step and tab, and the assignment is regenerated on every configuration update. See Detections.

FieldRequiredDefaultDescription
detection_rulesN-Detection rules to reduce against. Omit, or leave empty, to ship everything the pipelines produce
detection_rules[].nameYRule name, used to report which detections matched
detection_rules[].queryYThe rule's full source, inline. Not a reference to a rule stored elsewhere
detection_rules[].languageNkqlRule language: kql or yaral
unmatched_tablesNkeepWhat to do with an event no assigned detection is scoped to: keep or drop

An entry missing either name or query is skipped, and the skipped count is reported in the debug log. If every entry is skipped the smart pipeline does not run at all, which ships everything rather than nothing — reduction fails open throughout.

unmatched_tables accepts keep and drop case-insensitively. Any other value is treated as keep and reported as a warning, so a typo is visible rather than a silent no-op.

warning

drop discards every event that no assigned detection is scoped to. A detection scoped to a table only applies to events that declare that table, so events with no declared table are never matched by a table-scoped rule and are governed entirely by this setting. A detection that fails open on a join, union or lookup also applies to the tables it reads that way, so their events are kept rather than dropped. Vet the assigned rule set against your traffic before switching it.

Scheduling​

warning

Unlike most target types, sentineldatalake does not flush once per notification when left unconfigured. With neither interval nor cron set it flushes every 30 seconds.

See Scheduling and Pool Behavior for the interval, cron and max_items fields shared by all targets, and for how to override or opt out of this default.

Debug Options​

FieldRequiredDefaultDescription
debug.statusNfalseEnable debug logging
debug.dont_send_logsNfalseProcess logs but don't send to target (testing)

Details​

The Microsoft Sentinel data lake target provides cost-optimized ingestion for high-volume telemetry with extended retention requirements. Data lake ingestion offers significantly lower costs compared to standard DCR-based ingestion, making it ideal for firewall logs, DNS queries, network flows, and other high-fidelity telemetry requiring long-term storage.

Data Lake Benefits​

Cost Efficiency - Data lake ingestion costs are substantially lower than standard analytics ingestion, enabling cost-effective processing of massive telemetry volumes that would be prohibitively expensive with traditional methods.

High Fidelity - Preserves complete log detail without sampling or field reduction, maintaining full forensic capability for security investigations and compliance auditing.

Extended Retention - Optimized for long-term storage of high-volume logs, supporting retention periods spanning months or years for compliance requirements and historical analysis.

Director Proxy Integration​

The target supports two deployment models:

Direct Authentication - Director connects directly to Azure using service principal credentials (tenant_id, client_id, client_secret). This model requires Director to have network connectivity to Azure endpoints and credentials for the target subscription.

Director Proxy Forwarding - Director sends processed data to VirtualMetric Director Proxy (Azure Function) deployed in customer environment. Director Proxy uses Azure Managed Identity for credential-free access to Microsoft Sentinel data lake, eliminating the need to share Azure credentials with Director.

The Director Proxy model is particularly valuable for MSSP deployments where customers maintain complete control over Azure credentials while enabling centralized data processing and routing by the MSSP's Director infrastructure.

IAM Permissions​

The Sentinel data lake target uses the same authentication and ingestion pattern as the standard Sentinel target. When using Service Principal or Managed Identity authentication, the following Azure roles are required:

Azure RoleScopePurpose
Monitoring Metrics PublisherData Collection RuleSend data via DCR ingestion API
Monitoring ReaderResource GroupList DCEs and DCRs for stream auto-discovery

The Monitoring Reader role can also be assigned per resource instead of on the resource group. Autodiscovery works with either model, and the same scopes apply to the managed identity of a Director Proxy. Never assign it at subscription scope:

Azure RoleScopePurpose
Monitoring ReaderData Collection EndpointRead the DCE and find the DCRs linked to it
Monitoring ReaderEach Data Collection RuleRead the DCR's stream and table declarations
Monitoring Metrics PublisherEach Data Collection RuleSend data via DCR ingestion API

A DCR that is missing a role is left out of discovery and only its tables fail. See the Troubleshooting section of the Microsoft Sentinel target for the resulting errors.

If the setup functions (DCE/DCR/table creation) are used, additional roles are required:

Azure RoleScopePurpose
Monitoring ContributorResource GroupCreate and update DCEs and DCRs
Log Analytics ContributorLog Analytics WorkspaceCreate and update custom log tables

Stream Discovery​

When endpoint is specified as a Resource ID (not HTTPS URL), the target automatically discovers available Data Collection Rules and their associated streams. This autodiscovery feature simplifies configuration by eliminating manual stream enumeration.

While the target runs, Director reads the DCRs again every cache.timeout seconds (default 300), in the background, and applies your own streams list to the result as it does at startup. Sends never wait for that read: each worker picks up the new set of streams at its next send.

  • A stream that a new or newly readable DCR adds is used from the next send.
  • A stream whose DCR, endpoint, table or columns changed, or that is no longer discovered, is switched or dropped only after the records already buffered for it have been delivered. A refresh never discards a buffered record.
  • A refresh that fails, or that finds no streams, logs a Warning and keeps the streams from the last successful discovery.

The refresh is off when cache.timeout is 0 or less, or when endpoint is an https:// URL, which turns autodiscovery off.

Stream configurations can be filtered using the streams array to limit ingestion to specific tables. Each stream configuration supports independent DCR IDs via the rule_id field, enabling flexible routing to different data collection rules.

Field Management​

The target automatically detects table schemas and validates incoming data against defined columns. When keep_phantom_fields is false (default), fields not defined in the target schema are automatically removed before ingestion, preventing schema validation errors.

warning

Disabling keep_phantom_fields removes undefined fields. Ensure all required fields are included in your DCR schema.

Data is buffered per destination stream until batch_size records or buffer_size bytes are reached for that stream, or the flush schedule fires. Whichever limit binds first triggers the upload; the record that would have exceeded it rides the next one rather than being dropped. Because both limits are per stream, a target fanning out to several streams accumulates up to that much for each of them before a flush. The drop_unknown_stream_events setting (default: true) silently discards events for streams not configured in the target, preventing processing failures for unexpected data types.

With drop_unknown_stream_events: false, an event that maps to no configured stream is rejected instead, and logged as record rejected by target: no DCR stream for table "<table>" (stream "<stream>" is not configured on the target), or as no DCR stream for definition <N> for an event with no table. The target stays up and does not reinitialize. The payload is retried about every 2 seconds, and on its 4th delivery it is dropped (dropping (giving up)) and its undelivered events are added to the target's dropped counter. Events before the rejected one in the same batch are delivered, and events after it are dropped with it. So false does not hold data until the stream is fixed: it makes the drop visible, then drops the data all the same.

warning

Enabling drop_unknown_stream_events silently discards unmatched events. Monitor data flow to ensure expected streams are properly configured.

Field Normalization​

The field_format property normalizes log data to standard formats before ingestion:

  • csl - Common Security Log format
  • asim - Advanced Security Information Model

Normalization ensures consistent field naming and structure across diverse log sources, improving query efficiency and security analytics capabilities.

Examples​

Basic Configuration​

Minimum configuration using direct Azure authentication:

targets:
  - name: sentinel_data_lake
    type: sentineldatalake
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"

Managed Identity​

Using Managed Identity for credential-free authentication on Azure infrastructure:

targets:
  - name: managed_identity_data_lake
    type: sentineldatalake
    properties:
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"

Director Proxy​

Configuration using Director Proxy for credential-free forwarding:

targets:
  - name: proxy_data_lake
    type: sentineldatalake
    properties:
      function_app: "https://my-director-proxy.azurewebsites.net/api/Sentinel"
      function_token: "your-proxy-authentication-token"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"

Filtered Streams​

Configuration with specific stream filtering and custom settings:

targets:
  - name: filtered_data_lake
    type: sentineldatalake
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      streams:
        - name: "Custom-FirewallLogs"
        - name: "Custom-DNSLogs"
      keep_phantom_fields: false
      drop_unknown_stream_events: true
      cache:
        timeout: 600

High-Volume Processing​

Optimized configuration for high-volume log ingestion:

targets:
  - name: high_volume_data_lake
    type: sentineldatalake
    pipelines:
      - normalization
    properties:
      function_app: "https://my-director-proxy.azurewebsites.net/api/Sentinel"
      function_token: "your-proxy-authentication-token"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      batch_size: 5000
      field_format: "asim"
      streams:
        - name: "Custom-FirewallLogs"
          rule_id: "dcr-00000000000000000000000000000000"
        - name: "Custom-DNSLogs"
          rule_id: "dcr-11111111111111111111111111111111"

Debug Configuration​

Testing configuration with debug enabled:

targets:
  - name: debug_data_lake
    type: sentineldatalake
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "00000000-0000-0000-0000-000000000000"
      client_secret: "your-client-secret"
      endpoint: "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDCE"
      debug:
        status: true
        dont_send_logs: true  # Test mode - doesn't actually upload

Troubleshooting​

The sentineldatalake target authenticates, discovers streams, and uploads in exactly the same way as the sentinel target, so it reports the same errors for the same causes: missing Monitoring Reader or Monitoring Metrics Publisher roles, credential errors, an empty stream configuration, rejected uploads, and Director Proxy errors. Use the Troubleshooting section of the Microsoft Sentinel target for the full list of errors, causes, and fixes, and Target Delivery Errors for how Director logs and retries target failures.

One difference applies when reading that section for this target: log lines and the connection status carry this target's name, so match on the cause text, which is the part after Reason: or after the last colon, rather than on the target name shown in the examples. The errors in the Director Proxy table apply unchanged; the target wizard sets function_app and function_token through the same Proxy field as Microsoft Sentinel's.