Grafana Cloud

Observability

Synopsis

Creates a Grafana Cloud target that sends logs, metrics and traces to a Grafana Cloud stack over OTLP.

The target is the OTLP target with Grafana Cloud's sign-in and limits built in.

Schema

- name: <string>
  description: <string>
  type: grafanacloud
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    instance_id: <string>
    token: <string>
    send_metrics: <boolean>
    compression: <boolean>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    headers:
      <key>: <value>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be grafanacloud
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
url	Y	-	The stack's OTLP endpoint, such as https://otlp-gateway-prod-eu-west-2.grafana.net/otlp. It must use https. Grafana Cloud shows it in the stack's OpenTelemetry tile. In the console: OTLP endpoint
instance_id	Y	-	The stack's instance ID, a number, from the same tile. It can be plain text, an environment variable or a secret. In the console: Instance ID
token	Y	-	An access policy token that can write metrics, logs and traces. The tile can create one. It can be plain text, an environment variable or a secret. In the console: Access token
send_metrics	N	true	Sends the readings of metric devices as metrics. false sends them as log records. In the console: Readings as metrics, on by default
headers	N	-	Other headers, sent as they are with each request. YAML only

Performance

Field	Required	Default	Description
compression	N	true	Compresses each request with gzip. YAML only
timeout	N	30	How long a request may take, in seconds
batch_size	N	1000	The most events sent in one request
max_retries	N	3	How many times a failed request is retried
retry_delay	N	1	How long to wait before a retry, in seconds
disable_retries	N	false	true sends each batch once, as on the OTLP page. In the console: Disable retries

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

The debug options work as on the OTLP page.

Settings That Do Not Apply

The target always sends OTLP over HTTP with protobuf, and it checks Grafana Cloud's certificate against the system's trust store. So it ignores protocol, format, path, grpc_passthrough, address, port and tls.

It signs in itself, so it replaces an Authorization header set under headers.

The Console Form

The form has one step, Grafana Cloud settings:

• OTLP endpoint;
• Instance ID and Access token, each with the Manual / Environment variable / Vault picker;
• Readings as metrics;
• the Performance fields.

Details

Where to Find the Values

1. In the Grafana Cloud portal, open the stack.
2. On the OpenTelemetry tile, choose Configure.
3. The page shows the OTLP endpoint and the instance ID.
4. It can also generate an access token with the scopes the target needs: metrics:write, logs:write and traces:write.

How the Target Signs In

The target uses basic authentication, with the instance ID as the user name and the token as the password.

What Arrives Where

• Logs arrive in Loki, metrics in Mimir and traces in Tempo.
• The records of a device carry the device's name as service.name, unless a record carries a service.name of its own. Grafana Cloud uses it as the job label of metrics and the service_name label of logs.
• Metric names are converted to Prometheus style: dots become underscores, sums get _total, and units become suffixes. For example, a sum named system.io in bytes becomes system_io_bytes_total.

Device Readings as Metrics

They work as described in the OTLP page's Device readings as metrics section, with one difference.

Grafana Cloud's metrics store does not take delta sums unless delta ingestion is on for the tenant. So this target sends these readings as gauges:

• StatsD counters;
• CloudWatch Sum and SampleCount;
• Elasticsearch query counts and sums;
• libvirt and VMware counts for each interval.

In Grafana, sum such a gauge over time with sum_over_time, rather than taking its rate.

Size Limits

• Grafana Cloud refuses an OTLP request over 64 MB. The target keeps each request at or under 64,000,000 bytes, and splits a batch that would be larger.
• A single record over that limit is dropped and counted as dropped, with a warning in the Director log.
• A log line over 256 KB is sent as it is. Grafana Cloud refuses it and says so in its answer, and the target counts the line as dropped. The OTLP page's partial_success troubleshooting entry applies.

A Self-Managed Grafana Stack

There is no separate target for a self-managed Grafana stack (Loki, Mimir and Tempo). Use the OTLP target, pointed at Grafana Alloy or an OpenTelemetry Collector in front of the stack.

Why not straight to the backends. The OTLP target sends every signal to one address, while Loki, Mimir and Tempo each have their own. Alloy routes logs to Loki, metrics to Mimir and traces to Tempo.

Steps.

1. Point the OTLP target at Alloy's OTLP receiver. Alloy's ports are 4318 for HTTP and 4317 for gRPC.
2. Turn on Readings as metrics on the OTLP target, so device readings go as metrics.
3. Add otelcol.processor.deltatocumulative on Alloy's metrics path. Mimir, like Prometheus, refuses delta sums by default.

A minimal Alloy configuration to start from:

otelcol.receiver.otlp "default" {
  http {}
  grpc {}
  output {
    logs    = [otelcol.exporter.otlphttp.loki.input]
    metrics = [otelcol.processor.deltatocumulative.default.input]
    traces  = [otelcol.exporter.otlp.tempo.input]
  }
}

otelcol.processor.deltatocumulative "default" {
  output {
    metrics = [otelcol.exporter.otlphttp.mimir.input]
  }
}

otelcol.exporter.otlphttp "loki" {
  client {
    endpoint = "http://loki:3100/otlp"
    headers  = { "X-Scope-OrgID" = "<tenant>" }
  }
}

otelcol.exporter.otlphttp "mimir" {
  client {
    endpoint = "http://mimir:8080/otlp"
    headers  = { "X-Scope-OrgID" = "<tenant>" }
  }
}

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo:4317"
    tls {
      insecure = true
    }
  }
}

Loki and Mimir refuse a request that does not name a tenant in the X-Scope-OrgID header, unless multi-tenancy is turned off on them. Replace <tenant> with your tenant, or remove the headers lines if multi-tenancy is off.

Examples

A Grafana Cloud Stack

A Grafana Cloud stack, with the token from an environment variable...

targets: - name: grafana_cloud type: grafanacloud properties: url: "https://otlp-gateway-prod-eu-west-2.grafana.net/otlp" instance_id: "123456" token: "${GRAFANA_CLOUD_TOKEN}"

Device Readings as Log Records

Device readings as log records instead of metrics, with the token from a secret...

targets: - name: grafana_cloud_logs type: grafanacloud properties: url: "https://otlp-gateway-prod-eu-west-2.grafana.net/otlp" instance_id: "123456" token: "$secret{grafana-cloud-token}" send_metrics: false

Troubleshooting

Configuration Errors

Director refuses a configuration it cannot use, with one of these messages:

• url is required for Grafana Cloud
• url for Grafana Cloud must be the https OTLP endpoint from the stack's OpenTelemetry tile
• instance_id is required for Grafana Cloud
• token is required for Grafana Cloud

Errors from Grafana Cloud

The OTLP page's Troubleshooting entries apply. For Grafana Cloud in particular:

• status 401 or status 403: the instance ID or the token is wrong, or the token's access policy cannot write the signal (metrics:write, logs:write, traces:write).
• status 404: the url is not the stack's OTLP endpoint, or it already ends with /v1/logs. Set it to the address that ends in /otlp.
• status 429: the stack hit Grafana Cloud's rate limits. The target retries.
• [OTLP] partial_success warnings: Grafana Cloud refused some records, and the message names why, such as a log line over 256 KB or too many attributes.
• [OTLP] dropped a record of N bytes: Grafana Cloud refuses an OTLP <signal> request over 64000000 bytes: a single record was too large to send. <signal> is logs, metrics or traces. In the log, the warning ends with target= and the name of the target.