Azure Monitor Managed Service for Prometheus

Microsoft Azure Observability

Synopsis

Creates an Azure Monitor managed service for Prometheus target that sends metrics to an Azure Monitor workspace with remote-write.

It is the Prometheus remote-write target with Microsoft Entra sign-in built in. That page describes what the Prometheus targets have in common: what is sent, names and labels, counts over an interval, and the receiver's answers. This page covers only what differs.

note

This target sends metrics to an Azure Monitor workspace. To send logs to Log Analytics workspace tables, use the Azure Monitor Logs target instead.

Schema

- name: <string>
  description: <string>
  type: azprometheus
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
`name`	Y		Target name
`description`	N	-	Optional description
`type`	Y		Must be `azprometheus`
`pipelines`	N	-	Optional post-processor pipelines
`status`	N	`true`	Enable/disable the target

Connection

Field	Required	Default	Description
`url`	Y	-	The metrics ingestion endpoint of the workspace's data collection rule: `https://<endpoint>.<region>.metrics.ingest.monitor.azure.com/dataCollectionRules/<rule-id>/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24`. The workspace's Overview page in the Azure portal shows it. It must use https. The target uses it as it is set, with its query. In the console: Metrics ingestion endpoint

Authentication

Field	Required	Default	Description
`tenant_id`	N	-	The app registration's directory (tenant) ID. In the console: Tenant ID, when Sign-in is App registration
`client_id`	N	-	The app registration's application (client) ID. In the console: Client ID
`client_secret`	N	-	The app registration's client secret. In the console: Client secret

Set all three of tenant_id, client_id and client_secret to sign in as an app registration, or none of them to sign in as the host's identity (see Identities). Each can be plain text, an environment variable or a secret.

Performance

Field	Required	Default	Description
`timeout`	N	`30`	As on the Prometheus remote-write page
`batch_size`	N	`1000`	As on the Prometheus remote-write page
`max_retries`	N	`3`	As on the Prometheus remote-write page
`retry_delay`	N	`1`	As on the Prometheus remote-write page
`disable_retries`	N	`false`	As on the Prometheus remote-write page

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

The debug options work as on the OTLP page.

Settings That Do Not Apply

The target signs in itself, so it ignores username, password and token, and its token replaces an Authorization header set under headers. It also ignores the settings listed on the Prometheus remote-write page.

The Console Form

The form has one step, Azure Monitor managed service for Prometheus settings:

Metrics ingestion endpoint.
Sign-in, which is Managed identity (the default) or App registration. App registration shows Tenant ID, Client ID and Client secret, all required. The client secret has the Manual / Environment variable / Vault picker.
The Performance fields.

The Sign-in choice belongs to the form, not to the YAML. Managed identity sends none of the three fields, and App registration sends all three. The form has no TLS step, because Azure's endpoints use public certificates.

Details

The target asks Microsoft Entra ID for a token for https://monitor.azure.com/.default, and it puts the token on each request. The identity needs the Monitoring Metrics Publisher role on the workspace's data collection rule. The workspace's Overview page links to that rule.

A new role assignment can take about 30 minutes to work. Until then, the requests fail with 403.

Identities

Two ways:

An app registration: set tenant_id, client_id and client_secret.
The host's identity: leave out all three. The target then signs in as the managed identity of the Azure VM or scale set it runs on, or as its workload identity on Kubernetes, as the other Azure targets do. To pick a user-assigned managed identity, set AZURE_CLIENT_ID on the host to its client ID.

A client_id without a client_secret is refused, because the host's identity would not use it. A setting that is set but resolves to nothing stops the target from starting, so the target never signs in as another identity by mistake.

A failed token request is retried like a failed send. The token is never logged.

Not Supported Yet

Sovereign clouds, such as Azure Government and Azure China. They use other token scopes.

Names That Differ Only in Case

Azure Monitor ignores case in metric names, label names and label values, unlike open-source Prometheus. Two series that differ only in case are stored as one series, and a query can return either spelling. When both series send a sample with the same timestamp, Azure Monitor keeps one of the samples and drops the other. A sample whose labels include two names that differ only in case is dropped, because Azure Monitor reads them as the same label. Use one spelling for each name and value.

Everything Else

Everything else works as on the Prometheus remote-write page.

Examples

The Managed Identity of the Host

Signing in with the managed identity of the host...

targets:
  - name: azure_prometheus
    type: azprometheus
    properties:
      url: "https://my-dce-abcd.westeurope-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-0123456789abcdef0123456789abcdef/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24"

An App Registration

An app registration, with its secret from a secret store...

targets:
  - name: azure_prometheus_app
    type: azprometheus
    properties:
      url: "https://my-dce-abcd.westeurope-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-0123456789abcdef0123456789abcdef/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24"
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "11111111-1111-1111-1111-111111111111"
      client_secret: "$secret{azure-prometheus-client-secret}"

Troubleshooting

Configuration Errors

Director refuses a configuration it cannot use, with one of these messages:

url is required for Azure Monitor managed service for Prometheus
url for Azure Monitor managed service for Prometheus must be an https URL with a host, such as https://my-workspace.westeurope-1.metrics.ingest.monitor.azure.com/...
client_secret needs tenant_id and client_id: set all three, or none to sign in with the managed or workload identity of the host
client_id needs a client_secret: set the app registration's secret, or leave client_id out to sign in with the managed or workload identity of the host, ...: the message goes on to say that AZURE_CLIENT_ID on the host picks a user-assigned identity.
the client_secret of the Azure Monitor managed service for Prometheus target resolved to nothing: check the environment variable or the secret it names: the setting is set, but its environment variable or secret is empty. The same message names the tenant ID or the client ID.
failed to resolve the client_secret of the Azure Monitor managed service for Prometheus target: ...: the environment variable or the secret could not be read.

Errors While It Runs

[OTLP] token-error ...: Microsoft Entra ID gave no token. The tenant ID, the client ID or the client secret is wrong, the secret has expired, or the host has no managed identity. The target retries.
status 403: the identity lacks the Monitoring Metrics Publisher role on the data collection rule, or the role was assigned less than about 30 minutes ago. The batches are delivered again once this is fixed.
status 404: the data collection rule's ID or the path in url is wrong.
The warnings of the Prometheus remote-write page apply too.

Synopsis​

Schema​

Configuration​

Connection​

Authentication​

Performance​

Scheduling​

Debug Options​

Settings That Do Not Apply​

The Console Form​

Details​

Sign-in​

Identities​

Not Supported Yet​

Names That Differ Only in Case​

Everything Else​

Examples​

The Managed Identity of the Host​

An App Registration​

Troubleshooting​

Configuration Errors​

Errors While It Runs​