Skip to main content

Grafana Loki

Observability

Synopsis​

Creates a Grafana Loki target that sends logs to Grafana Loki through its push API. It works with a Loki that you run yourself and with Grafana Cloud Logs.

Schema​

- name: <string>
  description: <string>
  type: loki
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    format: <string>
    compression: <boolean>
    tenant_id: <string>
    headers:
      <key>: <value>
    authentication:
      type: <string>
      username: <string>
      password: <string>
      token: <string>
    labels:
      <label name>: <string>
    label_fields:
      <label name>: <string>
    structured_metadata:
      - <string>
    batch_size: <numeric>
    max_bytes: <numeric>
    timeout: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    keep_alive: <boolean>
    follow_redirects: <boolean>
    connect_timeout: <numeric>
    socket_timeout: <numeric>
    pool_max: <numeric>
    pool_max_per_route: <numeric>
    tls:
      status: <boolean>
      verify: <boolean>
      ca_name: <string>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      server_name: <string>
      min_tls_version: <string>
      max_tls_version: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration​

The following fields are used to define the target:

FieldRequiredDefaultDescription
nameYTarget name
descriptionN-Optional description
typeYMust be loki
pipelinesN-Optional post-processor pipelines
statusNtrueEnable/disable the target

Connection​

FieldRequiredDefaultDescription
urlY-The address of Loki, or of the gateway in front of it, such as https://loki.example.com. It must be an http or https URL with a host. A URL with no path, or with only /, gets the path /loki/api/v1/push. A URL with any other path is used as it is set. The target keeps the query of the URL in both cases. In the console: URL
formatNprotobufprotobuf or json. With protobuf, the body is a Snappy-compressed protobuf message, and the content type is application/x-protobuf. With json, the body is a JSON document, and the content type is application/json. YAML only
compressionNtrueApplies only to the json format. The target compresses the body with gzip and sends Content-Encoding: gzip. The protobuf format is always compressed with Snappy and has no Content-Encoding. YAML only
tenant_idN-The tenant. The target sends it in the X-Scope-OrgID header. Set it when your Loki has more than one tenant. In the console: Tenant ID (Optional)
headersN-A map of header names to values. The target sends each one on every request, as it is set. A header can hold a key, so the console withholds the values when it shows the target. The target sets some headers itself, and a header of the same name does not replace them (see Headers). YAML only

Authentication​

FieldRequiredDefaultDescription
authentication.typeNnoneThe sign-in: none, basic or bearer. The header type of the HTTP target is not available. In the console: Authentication method
authentication.usernameN-The user name for basic. It is required with basic. In the console: Username
authentication.passwordN-The password for basic. It is required with basic. In the console: Password
authentication.tokenN-The token for bearer. It is required with bearer. In the console: Token

username, password and token can each be plain text, an environment variable or a secret.

Labels​

FieldRequiredDefaultDescription
labelsN-The static labels, as a map from the label name to its value. In the console: Labels
label_fieldsN-The labels whose value the target reads from the record, as a map from the label name to the name of a field. In the console: Labels from record fields
structured_metadataN-A list of field names. The target sends each field it finds as structured metadata of the entry. The list has at most 128 fields. In the console: Structured metadata fields (Optional)

How the Labels Choose a Stream describes how the target works out the labels of each record.

Performance​

FieldRequiredDefaultDescription
batch_sizeN1000The number of records in one request. In the console: Batch size (Optional)
max_bytesN1048576The size, in bytes, at which the target sends a request. It is the batch size that Promtail uses. The length of each line plus 64 bytes counts toward it for each record. Director refuses a value below zero. YAML only
timeoutN60As on the HTTP page. In the console: Request timeout (Optional)
max_retriesN0The number of tries after the first one. -1 tries without limit. The target tries again only when the request never left the host (see How the Target Sends). In the console: Maximum retries (Optional)
retry_delayN1The number of seconds before the first retry. The later waits grow. In the console: Retry delay (Optional)
disable_retriesNfalseAs on the HTTP page. In the console: Disable retries

HTTP Client​

keep_alive (true), follow_redirects (true), connect_timeout (10), socket_timeout (10), pool_max (50) and pool_max_per_route (25) work as on the HTTP page's Request Settings and Connection Pool, with the same defaults. They are YAML only.

TLS​

The tls settings work as on the HTTP page. In the console: the TLS settings step.

Scheduling​

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options​

The debug options work as on the HTTP page.

The Console Form​

The form has four steps:

  • Grafana Loki settings. Under Connection settings: URL and Tenant ID (Optional). Under Performance: Batch size (Optional) and Request timeout (Optional), then the Disable retries switch, then Maximum retries (Optional) and Retry delay (Optional). The form hides the last two while Disable retries is on.
  • Authentication. Authentication method is None (the default), Basic or Bearer. Basic shows Username and Password. Bearer shows Token. The form sends only the fields of the chosen method.
  • Stream labels:
    • A notice, Every stream has two labels already. It says that the target labels each stream with service_name (the device's name) and device_type, that a label you add with one of these names replaces it, and that Loki takes at most 15 labels, these two included.
    • Labels, the labels with the same value on every line. It is a list with the columns Label name and Value, and an Add label button.
    • Labels from record fields, the labels whose value Director reads from the record. A field name with a dot reads a nested field, such as host.name. It is a list with the columns Label name and Record field, and an Add label from field button.
    • Structured metadata fields (Optional), which takes one tag for each record field.
  • TLS settings. The same fields as in the TLS settings step of the HTTP target.

The form checks the Stream labels step. In the messages below, <name> stands for the label name and N for a count:

  • A name must start with a letter from A to Z or an underscore, and use only letters from A to Z, digits and underscores. The messages are Label name is required, Label name must start with a letter from A to Z or an underscore, and use only letters from A to Z, digits and underscores and Label name cannot start with two underscores. A label also needs its value: Value is required under Labels, and Record field is required under Labels from record fields.
  • A name in both lists: "<name>" is already a label with a fixed value. Use each label name once.
  • A name twice in one list: "<name>" is in this list more than once. Use each label name once.
  • More than 15 labels: Loki takes at most 15 labels, service_name and device_type included. These settings add up to N.
  • More than 128 metadata fields: Loki takes at most 128 structured metadata fields. These settings list N.

The form has no fields for format, compression, max_bytes, headers, keep_alive, follow_redirects, connect_timeout, socket_timeout, pool_max and pool_max_per_route. They are YAML only. A target that has one of them in its YAML keeps it when you edit or clone the target in the console.

Details​

Where to Find the Values​

  • The URL: the address of your Loki, or of the gateway in front of it. For Grafana Cloud Logs, use the address of your Loki instance.
  • The tenant: set tenant_id when your Loki has more than one tenant. The target sends it in the X-Scope-OrgID header.
  • The sign-in: Grafana Cloud Logs takes basic. The user name is the user ID of your Loki instance. The password is an access policy token that is allowed to write logs. Keep the token in a secret.

How the Target Sends​

  • Each record becomes one line in a stream. The line is the whole record, as text.
  • The target sends the lines of a batch in one request. It sends the request when the batch holds batch_size records or reaches max_bytes.
  • The target tries a request again only when the request never left the host, such as when the connection could not be opened. This keeps one attempt from putting a batch in Loki twice.
  • When Loki answers with the status 400, 413 or 422, it has refused the batch for good. The target reports the batch as rejected. The HTTP page describes what Director does with a rejected batch.
  • Any other status outside 200 to 299 is an error. A wrong credential and a Loki that is down are among the causes. Director delivers the batch again later.

How the Labels Choose a Stream​

Loki keeps lines in streams. Records with the same labels go to the same stream. A record with different labels goes to another stream. The target works out the labels of each record like this:

  • Every record gets two labels by default. service_name is the name of its device. device_type is the type of its device.
  • labels adds static labels. The value is the same on every record.
  • label_fields adds labels whose value comes from the record. The target reads the field from the JSON of the record. It looks for the whole field name at the top level first. If that finds nothing, it follows a dotted name through nested objects, such as host.name. A string is the value of the label as it is. A number or a boolean gives its JSON text.
  • The target leaves a label out of a record when its value is empty. It also leaves the label out when the field is missing, is null, or is an object or an array.
  • A label of yours with the name of a default label replaces the default. The default does not come back for a record where your label has no value. That record has no label of that name.
  • The same name in both labels and label_fields is a configuration error.
  • A label name starts with a letter or an underscore, and holds only letters, digits and underscores. Loki keeps the names that start with __ for itself, so the target refuses them.
  • A stream has at most 15 labels. The count includes service_name and device_type unless you replace them.

A Record with No Label​

Loki refuses a stream with no labels, and it refuses the whole request with it. So a record that ends up with no label at all gets the label service_name="unknown_service". This is the name that Loki gives to a stream without a service. It can happen when you replace both default labels and the fields of your labels are missing from a record.

Why Few Different Label Values Matter​

Each different set of label values is one stream. A label whose value changes often makes many streams. Loki refuses new streams past its limit, which is 5,000 streams a tenant by default. The target logs a warning when it has sent more than 1,000 different streams in the last hour.

Use a label for a field that has few different values, such as an environment, a host or a level. Put every other field in structured metadata.

When to Use Structured Metadata​

Use structured metadata for a value that changes often, such as a trace ID. Structured metadata is not part of the stream, so its values can change without making new streams. In LogQL, you select a stream by its labels. You can then filter its lines on the structured metadata.

  • structured_metadata is a list of field names. The target reads each field as it does for label_fields. It sends the fields it finds as structured metadata of the entry.
  • The name of an entry is the field name with every character outside A-Z, a-z, 0-9 and _ replaced by _. So user.name becomes user_name. A name that starts with a digit gets an underscore in front.
  • Two fields that give the same name, such as trace.id and trace-id, are a configuration error.
  • Structured metadata needs Loki 3, or Loki 2.9 with it turned on. Older versions refuse it.

The Line, the Time and the Text​

  • The line of an entry is the whole record, as text.
  • The target replaces each run of bytes that is not valid UTF-8 with the character U+FFFD. It does this in the line, in the label values and in the structured metadata values, so that both formats send the same text. The target checks the limits on the cleaned text, because that is the text Loki gets.
  • The time of the entry is the time of the record. A record has no time when its epoch (the seconds since 1 January 1970) is zero or less. The target also treats a time that does not fit in a count of nanoseconds in 64 bits as no time. That is a time before 1678 or after 2262.
  • A record with no time gets the time at which the target gets the record. Both formats send the same time.

Limits​

The target keeps to the limits that Loki has by default:

  • A stream has at most 15 labels, counting service_name and device_type unless you replace them. The target refuses a configuration with more.
  • A label value has at most 2,048 bytes. The target refuses a configuration with a longer value in labels. It leaves a longer value that a field gives out of the labels of that record, and logs a warning at most once a minute.
  • A line has at most 256 KB. The target drops a longer record and counts it as dropped. It logs a warning at most once a minute.
  • structured_metadata lists at most 128 fields. The target refuses a configuration that lists more.
  • The names and values of the structured metadata of one entry total at most 64 KB. For a larger entry, the target sends the entry without its structured metadata, and logs a warning at most once a minute.
  • A tenant can have 5,000 streams by default. The target logs a warning at most once a minute when it has sent more than 1,000 different streams in the last hour. Labels whose values change often cause this. Move them to structured_metadata.
  • Loki refuses entries that are older than a week by default. The target does not change the time of a record, so Loki refuses a record that is older than that.

Sign-in, TLS and the HTTP Client​

The sign-in, the TLS settings and the HTTP client settings work as on the HTTP page, with these differences:

  • The sign-in takes only none, basic and bearer. basic sends the user name and the password. bearer sends the token in the Authorization header as Bearer <token>.
  • The target uses the tls settings only when tls.status is true. A client certificate needs its key, so set cert_name and key_name together or leave both out.
  • An https URL needs no tls settings when the host of Director trusts the certificate of Loki.

Headers​

The target sends each header in headers on every request, as it is set.

  • It sets X-Scope-OrgID from tenant_id and Authorization from authentication after the headers, so a header of the same name does not replace them. It sets X-Scope-OrgID only when tenant_id is set, and Authorization only for basic and bearer.
  • The content type and the content encoding follow from format and compression, and a header does not replace them either.
  • A header can hold a key, so the console withholds the values when it shows the target.

Examples​

A Loki with Tenants​

A Loki that has more than one tenant. Each stream gets the labels service_name and device_type...

targets:
  - name: loki
    type: loki
    properties:
      url: "https://loki.example.com"
      tenant_id: "tenant-1"

Grafana Cloud Logs​

Grafana Cloud Logs. The user name is the user ID of the Loki instance. The password is an access policy token, read from a secret...

targets:
  - name: loki_grafana_cloud
    type: loki
    properties:
      url: "https://<the address of your Loki instance>"
      authentication:
        type: basic
        username: "<the user ID of your Loki instance>"
        password: "$secret{store=corp-vault,ref=loki/access-policy-token}"

Labels and Structured Metadata​

Labels from fields, and a field that changes often as structured metadata. Each stream gets service_name, device_type, env, host and level. Each entry carries trace_id as structured metadata...

targets:
  - name: loki_labels
    type: loki
    properties:
      url: "https://loki.example.com"
      labels:
        env: "production"
      label_fields:
        host: "host.name"
        level: "level"
      structured_metadata:
        - "trace.id"

Troubleshooting​

Configuration Errors​

Director refuses a configuration it cannot use, with one of these messages. <name> stands for the name in your configuration and N for a number:

  • url is required for Loki
  • url for Loki must be an http or https URL with a host, such as https://loki.example.com: the URL has another scheme or no host. When the URL cannot be read at all, the same message continues with the reason.
  • format for Loki must be protobuf or json
  • label name "<name>" for Loki must start with a letter or an underscore and hold only letters, digits and underscores: the name breaks the rule. The same message appears for a name that starts with two underscores, which Loki keeps for itself.
  • label "<name>" for Loki is set in both labels and label_fields
  • label "<name>" for Loki has a value over 2048 bytes, which Loki refuses: the value is in labels. A value that a field gives is left out of that record instead, with a warning (see below).
  • the target has N labels for Loki, counting service_name and device_type, but Loki takes at most 15
  • structured_metadata lists N fields for Loki, but Loki takes at most 128 structured metadata entries
  • structured_metadata fields "<field>" and "<field>" for Loki both give the name "<name>": two fields give the same name for the entry, such as trace.id and trace-id, which both give trace_id.
  • batch_size must be greater than 0, got N
  • max_bytes must not be negative, got N
  • max_retries must be -1 (unlimited on transient errors) or >= 0, got N
  • username and password are required for basic authentication
  • token is required for bearer authentication
  • authentication type for Loki must be one of: none, basic, bearer
  • both tls.cert_name and tls.key_name must be provided together or omitted together

The two messages about basic and bearer appear when a credential is not set. Director reads each credential itself when the target starts.

Errors When the Target Starts​

  • failed to build TLS config: ...: the TLS settings could not be used. The rest of the message gives the reason.
  • failed to resolve http authentication <setting>: ...: the environment variable or the secret that a credential names could not be read. <setting> is username, password or token. The target does not start.

Warnings While It Runs​

These appear in the Director log. In the log, each warning ends with target= and the name of the target. That ending is left out below. N is the count since Director last logged that warning. Director logs each of these warnings at most once a minute:

  • [Loki] dropped N records in the last minute: Loki refuses a line over 262144 bytes: the target dropped records whose line was over 256 KB, and counted them as dropped.
  • [Loki] left out N label values over 2048 bytes in the last minute: Loki refuses them: a field gave a label value over 2,048 bytes. The target left that label out of the record.
  • [Loki] sent N entries without their structured metadata in the last minute: it was over 65536 bytes: the names and values of the structured metadata of an entry were over 64 KB. The target sent the entry without its structured metadata.
  • [Loki] wrote more than 1000 streams in the last hour: Loki refuses new streams past its limit, 5000 a tenant by default, so check the labels for values that change often: the labels make many streams. Move the fields whose values change often to structured_metadata.

Errors While It Runs​

Director logs one of these messages when a request fails. A message never holds the URL, because a URL can carry a credential. The first part wraps a reason, as in failed to send logs to Loki (not retried): received status code 401 from Loki: ....

  • failed to send logs to Loki after N retries: ...: the request never left the host on any try, such as when the connection could not be opened. N is max_retries. The rest of the message gives the last reason. Director delivers the batch again later.
  • failed to send logs to Loki (not retried): ...: the request may have reached Loki, so the target does not send it again itself. The rest of the message gives the reason. Director delivers the batch again later.
  • failed to send logs to Loki: shutdown during retry (attempt N): ...: the target or Director stopped while the target waited to try again. The rest of the message gives the last reason.
  • failed to create request: ...: the target could not build the request. The rest of the message gives the reason.
  • failed to send request: ...: the request failed on its way to Loki, such as a connection that could not be opened or an answer that did not arrive in time. The rest of the message gives the reason.
  • received status code N from Loki: ...: Loki answered with a status outside 200 to 299. After the colon comes the start of Loki's answer, at most 512 bytes. A wrong credential and a Loki that is down are among the causes.
  • record rejected by target: received status code N from Loki: ...: N is 400, 413 or 422. Loki refused the batch for good, and the target reports the batch as rejected.