Prometheus Remote-Write

Synopsis

Creates a Prometheus remote-write device. It makes Director the endpoint that Prometheus, Grafana Alloy and the OpenTelemetry Collector push their metrics to. You point the senders at the device, and they write their samples to it.

The device reads Prometheus remote write 1.0, the message all three senders can send. It does not read remote write 2.0, and it answers a sender that uses it with 415. Each sample in a request becomes one metric record. The record has the shape of the records that the Prometheus device writes when it scrapes, so a pipeline and a target handle both in the same way.

This device receives pushed metrics. The Prometheus device scrapes them, and the Prometheus remote-write target sends them out.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: prometheus_remote_write
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <numeric>
    authentication: <string>
    username: <string>
    password: <string>
    token: <string>
    read_timeout: <numeric>
    idle_timeout: <numeric>
    max_connections: <numeric>
    access_list:
      - action: <string>
        ip_blocks: <string>
    tenants:
      - tenant_id: <string>
        ip_blocks: <string>
    max_buffer_size: <string>
    tls:
      status: <boolean>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      min_tls_version: <string>
      max_tls_version: <string>
      client_ca_name: <string>
      client_auth_required: <boolean>

Configuration

The following fields are used to define the device. No field under properties is required, because every field has a default. The credentials are the one exception: authentication basic needs a username and a password, and authentication bearer needs a token.

Device

Field	Required	Default	Description
`id`	Y	-	Unique numeric identifier
`name`	Y	-	Device name
`description`	N	-	Optional description
`type`	Y	-	Must be `prometheus_remote_write`
`tags`	N	-	Optional tags
`pipelines`	N	-	Optional pre-processor pipelines
`status`	N	`true`	Enable/disable the device

Listener

Field	Required	Default	Description
`address`	N	`0.0.0.0`	The address to listen on. The default is every interface. The console takes an IPv4 address. In the console: Listen address (Optional)
`port`	N	`9090`	The port to listen on. It is a whole number from 1 to 65535. 9090 is the port of a Prometheus server. Any other port works too, because each sender is told the address and the port. In the console: Port
`read_timeout`	N	`30`	How long a request may take to be read, headers and body. It is a number of seconds, or a duration of whole seconds such as `5m`, from 1 to 3600. A request may also wait up to 30 seconds in all for its memory reservation and for the ingest buffer. That wait does not count against `read_timeout`. After both, the device has 30 more seconds to write its answer. YAML only
`idle_timeout`	N	`120`	How long a kept-alive connection may wait for its next request. It is from 1 to 86400 seconds, written like `read_timeout`. YAML only
`max_connections`	N	`10000`	How many connections may be open at once, from 1 to 100000. The device closes a connection past the limit as it accepts it. YAML only

Authentication

Field	Required	Default	Description
`authentication`	N	`none`	How senders sign in: `none`, `basic` or `bearer`, in any case. With `none`, the device lets every sender in, so use it only on a network you trust. In the console: Authentication method
`username`	Y with `basic`	-	The user that a sender signs in as. It is plain text, `${VARIABLE}` or `$secret{...}`. The device removes blanks around it. It cannot hold a colon, because basic sign-in splits the user and the password at the first colon. In the console: Username
`password`	Y with `basic`	-	The password that a sender signs in with. It is written like `username`. The device uses it as written, blanks included. In the console: Password
`token`	Y with `bearer`	-	The token that a sender signs in with. It is written like `username`. The device removes blanks around it. In the console: Token

The device refuses a username, a password or a token that does not fit authentication. It refuses basic without both a username and a password, and bearer without a token. It also refuses a credential that resolves to nothing.
The device checks every value when it reads the configuration. A configuration that cannot run stops the device with an error before it binds a port. The errors are under Configuration Errors.

See Signing In for what a sender sends.

Access List and Tenants

Field	Required	Default	Description
`access_list`	N	-	The source-IP access rules: a list of rules, each with an `action` (`accept` or `drop`) and `ip_blocks` (addresses or CIDR ranges). The device closes a connection from a source that the list refuses as it accepts it, before its TLS handshake. YAML only
`tenants`	N	-	The tenant map: a list of entries, each with a `tenant_id` and `ip_blocks`. A request from a source IP mapped to a tenant carries that tenant. YAML only

TLS

Field	Required	Default	Description
`tls.status`	N	`false`	Turns TLS on, so that the device serves HTTPS. In the console: TLS encryption
`tls.cert_name`	Y*	`cert.pem`	The certificate that the device serves. The console marks it as required. In the console: Certificate
`tls.key_name`	Y*	`key.pem`	The private key of the certificate. In the console: Private key
`tls.passphrase`	N	-	The passphrase of the private key, when the key has one. In the console: Passphrase (Optional)
`tls.min_tls_version`	N	`tls1.2`	The lowest TLS version that a connection may use. The console offers TLS 1.0, TLS 1.1, TLS 1.2 and TLS 1.3. In the console: Minimum TLS version (Optional)
`tls.max_tls_version`	N	-	The highest TLS version that a connection may use. Leave it out to allow up to TLS 1.3. In the console: Maximum TLS version (Optional)
`tls.client_ca_name`	N	-	The CA that signed the certificates the senders present. It must be a CA certificate in PEM form. The device checks any certificate that a sender presents against it, whether or not one is required. In the console: CA certificate
`tls.client_auth_required`	N**	`false`	Makes the device refuse a sender that presents no certificate. In the console: Client certificate verification

* = Required when tls.status is true.

** = Requires tls.client_ca_name.

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
Inline PEM content — used verbatim when the value contains -----BEGIN.
Environment variable — ${ENV_VAR}.
Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

Ingest Buffer

Field	Required	Default	Description
`max_buffer_size`	N	`128MiB`	The ingest admission cap, as on the other device pages. The minimum is `16MiB`. A value that is not positive is ignored: it does not mean unlimited. YAML only

This device has one stream, for its metric records. When Director runs without crash_resistance (memory mode), the device holds up to 3 times the value: 384 MiB at the default. See Listener Memory Sizing.

Pointing the Senders at the Device

The device answers POST /api/v1/write, the remote write path of Prometheus. It also answers POST /api/v1/push, for the senders of Mimir and Cortex.

Use https:// in the sender's URL when the device has TLS on.
Use http:// only when it has none, on a network where nobody else can read the traffic.

Prometheus, in prometheus.yml:

remote_write:
  - url: https://director.example.com:9090/api/v1/write
    # Remote write 1.0, the default message
    protobuf_message: prometheus.WriteRequest
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/remote-write-token
    tls_config:
      ca_file: /etc/prometheus/director-ca.crt
    # Metadata gives each metric its type. It is sent by default, once a minute
    metadata_config:
      send: true
      send_interval: 1m

For a username and a password, use basic_auth with username and password_file in place of authorization.

Grafana Alloy:

prometheus.remote_write "director" {
  endpoint {
    url = "https://director.example.com:9090/api/v1/write"

    bearer_token_file = "/etc/alloy/remote-write-token"

    tls_config {
      ca_file = "/etc/alloy/director-ca.crt"
    }

    metadata_config {
      send          = true
      send_interval = "1m"
    }
  }
}

OpenTelemetry Collector, with the prometheusremotewrite exporter:

extensions:
  bearertokenauth/director:
    filename: /etc/otelcol/remote-write-token

exporters:
  prometheusremotewrite:
    endpoint: https://director.example.com:9090/api/v1/write
    auth:
      authenticator: bearertokenauth/director
    tls:
      ca_file: /etc/otelcol/director-ca.crt

service:
  extensions: [bearertokenauth/director]
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheusremotewrite]

Signing In

With basic, a sender sends Authorization: Basic <user and password>, as the basic_auth setting of Prometheus does.
With bearer, a sender sends Authorization: Bearer <token>, and the word Bearer can be in any case. In Prometheus that is authorization with type: Bearer.
A request that sends nothing, or the wrong thing, gets 401. The answer carries WWW-Authenticate: Basic realm="VirtualMetric", charset="UTF-8" for basic. It carries WWW-Authenticate: Bearer realm="VirtualMetric" for bearer.
Turn TLS on when senders sign in. Without TLS, the device warns when it starts, because the credentials travel in the clear.

Answers

The device takes POST requests on /api/v1/write and /api/v1/push. It checks a request in this order: the source (the access list), the path, the method, the sign-in, the headers, the size of the body, and then the body.

Answer	When
`204`, with an empty body	The device stored every record of the request. A `204` does not mean that no sample was dropped (see What the Device Leaves Out). The device also answers `204` and drops the samples while Director routes nothing
`400`	The body is not a Snappy block, or what it holds is not a valid message. The device writes nothing of such a request
`401`	The sender did not sign in, or signed in with credentials the device does not take. The answer carries a `WWW-Authenticate` header (see Signing In)
`403`	An edit of the access list denied the source after its connection was open. The device closes the connection. A source that the list denies when it connects has its connection closed as the device accepts it, so it gets no answer
`404`	The path is not `/api/v1/write` or `/api/v1/push`
`405`	The method is not `POST`. The answer carries `Allow: POST`
`413`	The body is larger than 10 MiB as it is sent. Or the length that opens the Snappy block is larger than 64 MiB
`415`	The `Content-Type` is not `application/x-protobuf`. Or its `proto` is not `prometheus.WriteRequest` (remote write 2.0 is `io.prometheus.write.v2.Request`). Or the `Content-Encoding` is not `snappy`
`500`	The device failed on the request
`503`	Director cannot take the samples now: the ingest buffer stayed full for 30 seconds, storing the records failed, the memory for requests stayed full for 30 seconds, or the device is stopping. The answer carries `Retry-After: 30`, except while the device is stopping. The sender sends the request again

Every error answer has a plain-text body, not JSON. The exact messages are under What the Sender Sees.

The Headers and the Body

A request that has no Content-Type is read as remote write 1.0, and one that has no Content-Encoding is read as Snappy. Prometheus's own receiver does the same. The device answers any other encoding, identity included, with 415.
The body is a Snappy block (not the framed format) that holds a write request. The device refuses a block that claims more than 64 MiB without decoding it.

Limits

Memory. The device reserves memory for each request it handles. All its requests together may reserve at most 256 MiB.
- A request reserves memory for its body before the device reads it, so the bodies that are still arriving count too. A body without a Content-Length header reserves 10 MiB until it has arrived, and then gives back what it did not use.
- Before the body is decoded, the request also reserves memory for the decoded body.
- When the 256 MiB are taken, a request waits up to 30 seconds for its reservation, and then gets 503. The wait does not count against read_timeout: the time the sender has to send its body starts once the reservation is made.
Time. read_timeout, idle_timeout and max_connections limit the requests and the connections (see Listener).
A 503 can write a sample twice. The samples that the device wrote before the failure stay written. The sender sends the whole request again, as Prometheus does for every 5xx answer, so those samples are written again.
While Director routes nothing, the device checks the headers and the size of a request, drops its body without decoding it, and answers 204. This happens after the sender signs in. Such a request reserves no memory and never waits for a reservation. Its read_timeout counts from the start of the request, headers included.

Records

The device writes one metric record for each sample, as JSON. The records have the shape of the samples of the Prometheus device:

Field	Value
`@timestamp`	The time of the sample. A sender gives it in milliseconds. A time that no record can hold, a year past 9999, is replaced by the time the device read the request
`ecs.version`	`8.11.0`
`event.module`	`prometheus`
`event.dataset`	`prometheus.metric`
`event.kind`	`metric`
`event.category`, `event.type`	`["host"]`, `["info"]`
`prometheus.metric.name`	The `__name__` label of the series
`prometheus.metric.type`	`counter`, `gauge`, `histogram`, `gaugehistogram`, `summary`, `info`, `stateset` or `untyped` (see How the Device Finds the Type of a Metric)
`prometheus.metric.value`	The value. The device always writes it with a decimal point or an exponent (`3.0`, never `3`), so that a reader does not take it for an integer
`prometheus.metric.unit`	The unit from the metadata. The device leaves it out when the unit is not known
`prometheus.labels`	Every label of the series except `__name__`. A label given twice keeps the first. The device leaves the object out when there are no labels
`source.ip`	The address of the connection the request came over, never a header that a sender writes. An IPv4 address that a dual-stack socket reads as IPv6 is written as IPv4
`observer`	`vendor` is `VirtualMetric`, `product` is `Prometheus remote-write`, `name` is the device's name, and `hostname` is the Director's

Each record also carries the address of the client, and its tenant when the tenant map names one. They are not fields of the JSON.

A record looks like this:

{
  "@timestamp": "2026-10-30T09:15:00.123Z",
  "ecs": { "version": "8.11.0" },
  "event": {
    "module": "prometheus",
    "dataset": "prometheus.metric",
    "kind": "metric",
    "category": ["host"],
    "type": ["info"]
  },
  "observer": {
    "vendor": "VirtualMetric",
    "product": "Prometheus remote-write",
    "name": "prometheus-remote-write",
    "hostname": "director-01"
  },
  "source": { "ip": "10.0.4.17" },
  "prometheus": {
    "metric": {
      "name": "http_requests_total",
      "type": "counter",
      "value": 1027.0
    },
    "labels": {
      "instance": "web-01:9100",
      "job": "web",
      "method": "GET"
    }
  }
}

How the Device Finds the Type of a Metric

Prometheus does not send the type in a sample. It sends it in metadata, in a request of its own, once a minute by default. The device keeps the metadata of each metric family in memory, and it types each sample from it.

The device looks for a family with the exact name of the series.
If there is none, it removes one of the suffixes _bucket, _sum, _count, _gsum, _gcount, _total or _created from the name and looks again. A histogram latency_seconds types latency_seconds_bucket.
A series with no family, and a family whose type is unknown, is untyped.

A request that holds both series and metadata types its own series with its own metadata.
The device keeps the metadata of 100000 families at most. Once it is full, the samples of a new family are written as untyped, and the device logs a warning at most once a minute. The metadata of a family it already holds is still updated.
The device ignores a family whose name or unit is longer than 1024 bytes.
The device forgets the metadata when it restarts. Its samples are untyped until the sender sends the metadata again, which Prometheus does within send_interval.

What the Device Leaves Out

The device still answers 204 for the rest of the request when it drops one of these:

What	Why
A sample whose value is NaN or infinite	JSON holds neither. A stale marker, which Prometheus sends when a series goes away, is a NaN. This is routine, so the device logs it at the debug level
A native histogram	The device reads samples with a float value only. It logs a warning
A sample of a series that has no `__name__` label	The record would have no metric name
A sample whose record is larger than the ingest buffer admits	The device logs a warning (see Ingest Buffer)
An exemplar	The device does not read exemplars. It does not count them

With a route in place, the device counts the samples of every request that it decodes and answers 204: how many it received, how many it wrote, and how many it dropped for each reason above. A request that cannot be decoded counts as one request, because its samples are not known.
With no route, the device answers 204 without decoding the request, and it counts nothing.
The device logs each kind of drop at most once a minute while it goes on, and it logs the totals when it stops.

Sending the Data On

A target that sends metrics, such as the OTLP target, reads these records as it reads the records of the Prometheus device:

It sends a counter as a counter.
It also sends the _bucket, _sum and _count samples of a histogram, and the _sum and _count samples of a summary, as counters.
It sends every other sample as a gauge, the quantiles of a summary among them.

Changing the Configuration

The device starts again when you change the address, the port, the TLS settings, authentication, a credential, a timeout or the connection limit, and when Director starts or stops routing anything. It also starts again for a password that was rotated in a vault.
A secret that Director cannot read just now does not restart the device.
An edit that no longer validates starts the device again too, and the start reports why it cannot run.
An edit of access_list or tenants reaches the running device at once. A device that started without one starts again when you add the first one.

Clusters

Every node of a cluster, and every replica of a container Director, runs the device and binds the port. A sender pushes to one node, or to the load balancer in front of the nodes.

The device is not simulated, because its records are the samples that its senders push.

Console

In the console, go to Fleet Management > Devices > Prometheus remote-write, listed under Protocols, and click Add new device. The wizard's steps are:

General settings, as on every device.
Protocol settings. The step's description says: Prometheus servers, Grafana Alloy, the OpenTelemetry Collector and other remote-write senders write their samples to this director's address and port, as they would to Prometheus or any other remote-write receiver. The director writes each sample as a metric record.
Review. It shows the address (0.0.0.0 when the field is blank), the port, the authentication method, the username and the password or the token, and the TLS settings. It shows the password, the token, the certificate, the private key and the CA certificate as a mask when they are set and as a dash when they are not, never as themselves.

The Protocol settings step has these fields:

Listen address (Optional) and Port share one row. The placeholder of the address is 0.0.0.0, and its helper text says: The address the listener binds. Defaults to 0.0.0.0, every interface. The placeholder of the port is Enter the port, and its helper text says: Senders write to this port. Enter a whole number from 1 to 65535. Defaults to 9090.
Under Authentication, described as How senders sign in before they may write. None lets any sender write. Basic takes a username and a password, and Bearer a token, each as plain text, an environment variable or a vault secret.: the dropdown Authentication method. It offers None, Basic and Bearer, and None is the default. Its helper text says: None lets any sender write. Basic and Bearer are the two ways a Prometheus sender can sign in.
With Basic: Username and Password share one row. The placeholder of the username is e.g. prometheus, and its helper text says: Senders sign in as this user when they use Basic authentication. The placeholder of the password is Enter the password, and its helper text says: Senders sign in with this password when they use Basic authentication.
With Bearer: Token. Its placeholder is Enter the token. Its helper text says: Senders sign in with this token when they use Bearer authentication.
The console sends only the credentials of the chosen method, because Director refuses a device that carries the credentials of another method. On a saved device, a password or a token that you leave as it was keeps its stored value.
A warning notice, The credentials are sent in cleartext. It shows while TLS encryption is off and senders sign in: with Basic once the username is filled in, and with Bearer once the token is. It stays hidden while an environment variable may turn TLS on. Its subtitle says: Over plain HTTP, senders send their username and password, or their token, unencrypted with every write, and anyone who reads them can write to the device. Turn TLS encryption on, or sign in this way only where the network between the senders and the director is trusted.
Under TLS settings: the TLS encryption switch, which is off by default. It is Disabled or Enabled. Its helper text says: TLS encrypts data between senders and the director. Senders can also be required to present their own certificates. Without TLS, data travels over unencrypted HTTP.

With TLS encryption on, the fields come in this order:

An info notice, Certificate chain and trust requirements. It says: If an intermediate CA issued the certificate, include the full chain: the certificate first, then each intermediate. The other end of the connection must trust the issuing CA, so install the CA certificate on machines that do not trust it yet.
Certificate, with an upload button.
Private key, with an upload button.
Passphrase (Optional), with the placeholder Enter the passphrase.
The Client certificate verification checkbox. Its helper text says: Enables mutual TLS: every client must present a certificate that verifies against the CA below.
While it is checked, CA certificate.
Last, Minimum TLS version (Optional) and Maximum TLS version (Optional) in one row. Both show Select a TLS version as the placeholder. The helper text of the minimum says: The lowest TLS version the connection may use. Defaults to TLS 1.2. The helper text of the maximum says: The highest TLS version the connection may use. Leave empty to allow up to TLS 1.3. On the device page, an empty minimum reads Default (TLS 1.2).

The certificate and the private key use the TLS layout that the other listener devices share. Each takes the text, an environment variable, a Vault secret or a file path.

tls.status and tls.client_auth_required can each hold an environment variable in the YAML:

For TLS encryption, the form then shows the variable in a text field in place of the switch. Its placeholder is e.g. ${PROMETHEUS_SETTING}, and its helper text says: The director reads true or false from this environment variable. The console cannot show the value. Clear the field to use the switch instead.
For Client certificate verification, the checkbox stays, and a text field of the same name shows the variable after the TLS version fields. It has the same placeholder, and its helper text says: The director reads true or false from this environment variable in place of the switch above. The console cannot show the value. Clear the field to use the switch instead. The console saves the variable, not the checkbox, until the field is cleared.

The form checks the step against the rules that Director applies when it starts. In the messages below, <value> stands for what the field holds and <size> for a size:

The address: Listen address must be an IPv4 address, or 0.0.0.0 for every interface.
The port: Port is required, and Port must be a whole number from 1 to 65535.
The authentication method: Authentication method must be None, Basic or Bearer.
With Basic: Username is required for Basic authentication, and Password is required for Basic authentication.
With Basic, a typed username that contains a colon: Username must not contain a colon for Basic authentication.
With Bearer: Token is required for Bearer authentication.
A switch that holds something other than blank or an environment variable: The director reads this switch as on or off, not as <value>: clear the field to choose with the switch, or name an environment variable such as ${PROMETHEUS_SETTING}.
With TLS encryption on, the certificate: Certificate is required when it is empty, Provide a valid certificate to continue when the text is not a certificate, This certificate has expired. Provide a valid certificate to continue, and This certificate is larger than <size>. Provide a smaller certificate to continue.
With TLS encryption on, the key: Private key is required.
The TLS versions: TLS version must be TLS 1.0, TLS 1.1, TLS 1.2 or TLS 1.3, and Maximum TLS version must not be lower than Minimum TLS version (TLS 1.2 when empty).
Client certificates: Provide a CA certificate to verify client certificates against. It shows when client certificates are required and no CA is given.

The form checks only a username that you type. A username that comes from an environment variable or a vault secret is read by Director when the device starts, and Director still refuses one that contains a colon.

The form has no fields for read_timeout, idle_timeout, max_connections, access_list, tenants and max_buffer_size. They are YAML only. A device that has read_timeout, idle_timeout or max_connections in its YAML keeps them when you edit the device in the console.

Troubleshooting

Configuration Errors

The Director refuses a configuration it cannot use, with one of these messages. <name> stands for the name of the device, <key> for the name of a field, <value> for what the field holds, <reason> for the reason the Director gives and N for a number. The log line is Can not validate listener config for <name>. Error: <message>, and the connection state of the device says Configuration validation failed for <name>: <message>. The device does not start:

address not valid: the address cannot be used. The Director checks the port first, so a bad port is reported as one.
<key> is not a whole number: <key> is port, read_timeout, idle_timeout or max_connections.
port must be from 1 to 65535, not N
read_timeout must be from 1 to 3600, not N
idle_timeout must be from 1 to 86400, not N
max_connections must be from 1 to 100000, not N
<key> is <value>: it is true or false: <key> is tls.status or tls.client_auth_required. The switch holds a value that is neither true nor false.
tls: the certificate or the private key does not load
tls: the private key does not decrypt with its passphrase: <reason>
tls: the certificate and the private key are not a pair: <reason>
tls: min_tls_version is above max_tls_version
tls.client_auth_required needs tls.client_ca_name: the CA that signed the certificates writers present
tls.client_ca_name names no CA certificate that loads
tls.client_ca_name holds no PEM certificate: it must be a CA certificate in PEM form
authentication must be none, basic or bearer, not "<value>"
token is how a sender signs in with authentication bearer: with basic a sender signs in with a username and a password
authentication basic needs a username and a password: a sender signs in with both
password is set without a username: a sender signs in with both
username is set without a password: a sender signs in with both
the username or the password resolves to nothing: senders would sign in with an empty one
the username holds a colon: with basic authentication the username of a sender ends at its first colon, so no sender could sign in with this one
username and password are how a sender signs in with authentication basic: with bearer a sender signs in with a token
authentication bearer needs a token: a sender signs in with it
the token resolves to nothing: senders would sign in with an empty one
a username, a password or a token is set, but authentication is none: set authentication to basic or bearer, or remove them
a secret could not be resolved: the <what> of device "<name>": <reason>: a variable that is not set, or a vault that does not answer. <what> is username, password or token. For TLS, the message reads a secret could not be resolved: the TLS <what> of device "<name>": <reason>, and <what> is certificate, private key, passphrase or clients' CA.

Errors When the Device Starts

These appear in the Director's log. The device does not start.

Can not start PrometheusRemoteWrite listener for <name>. Error: <reason>: the port is taken, or the TLS files do not load. The connection state of the device says Connection validation failed for <name>: <reason>.
Can not start the listener of <name>. Error: <reason>: the device could not bind its address. The connection state of the device says Can not start the listener of <name>: <reason>.
Failed to run the PrometheusRemoteWrite listener of <name>. Error: <reason>: the server stopped with an error.
Skipping initialization of listener.: the device is turned off, so the Director does not start it. It is an information line.
senders sign in over HTTP without TLS: their credentials travel in the clear (turn TLS on) (<name>): a warning. authentication is basic or bearer and TLS is off. The Director logs it once, when the device starts.

Log Lines While It Runs

These appear in the Director's log. <name> stands for the name of the device, <client> for the address the request came from, <status> for a status code and N for a count. The Director logs each line below at most once a minute for each device and each kind of line. A refusal counts for each status separately. When it holds lines back, the next line ends with (N more like it since this was last logged). The lines of the start, the stop and the errors of the start are not held back.

Information:

Listener (<name>) server is successfully started on <address> over <http or https>, serving Prometheus remote write <without signing in, or with senders signing in (<basic or bearer>)>. <address> is the address and the port, such as 0.0.0.0:9090.
Listener (<name>) took N samples: N written, and dropped N with a value that is not finite, N native histograms, N without a metric name and N too large for the ingest buffer. N requests could not be decoded. The Director logs it when the device stops.

Warnings:

Closed a connection from <client> to <name> as it was accepted: the access list denies it.
Closed a connection from <client> to <name>: N connections are open, as many as max_connections takes. N is the value of max_connections.
Refused a request from <client> to <name>: it did not sign in.
Refused a request from <client> to <name>: it signed in with credentials the device does not take.
Refused a write from <client> to <name> with status <status>: <message>. <status> is 400, 413 or 415, and <message> is the text the sender gets (see What the Sender Sees).
A write from <client> to <name> waited 30s for room among the requests being served, which may hold 256 MiB of bodies at once, and was answered 503 to be sent again.
A write from <client> to <name> waited 30s for the device's ingest buffer, and was answered 503 with N of its samples written, which a retry writes again.
Ignored the metadata of N metric families sent to <name>: it keeps the metadata of 100000 families, as many as it takes, so the samples of a new family are written as untyped.
Dropped N native histograms of a write from <client> to <name>: the device reads samples with a float value only.
Dropped N samples of a write from <client> to <name>: their series has no __name__ label.
Dropped N samples of a write from <client> to <name>: each makes a record larger than the device's ingest buffer admits (max_buffer_size).
The listener of <name> did not stop within 10s: the writes it still waits on are dropped.
<name> did not stop within 30s; its collector waits for it.

Errors:

Can not write the samples of a write from <client> to <name>, N of them written, and answered it 503. Error: <reason>
Can not write the samples of a write from <client> to <name> to disk, N of them written. The write was answered 503, which a retry writes again. Error: <reason>
A request from <client> to <name> failed: <reason>. Stack: <stack>: the device answered 500.
PrometheusRemoteWrite heartbeat check failed. Stopping collector...: the device stopped answering its health check, and the Director stops it. The connection state of the device says PrometheusRemoteWrite heartbeat check failed for <name>.
Failed to update device configuration for <name>: the Director could not read the updated configuration of the device.

Debug:

Skipped N samples of a write from <client> to <name> whose value is NaN or infinite, stale markers among them.
The HTTP server of <name>: <message of the HTTP server>

What the Sender Sees

A 204 has an empty body. Every error answer has a plain-text body, not JSON. The messages are:

Answer	Message
`400`	`the body could not be read: <reason>`; `the body is not a Snappy block: <reason>`; and `the request is not a Prometheus write request: <reason>`
`401`	`unauthorized`
`403`	`forbidden`
`404`	`not found`
`405`	`method not allowed`
`413`	`the body is larger than 10 MiB`; or `the body is larger than 64 MiB once it is decompressed`
`415`	`unsupported Content-Type: remote write is application/x-protobuf`; or `unsupported remote write version: the device reads remote write 1.0 only, proto=prometheus.WriteRequest`; or `unsupported Content-Encoding: the body must be a Snappy block (snappy)`
`500`	`internal error`
`503`	`the director is busy: write again later`; or `the director could not store the samples: write again later` (both carry `Retry-After: 30`); or `the device is stopping: write again later` (no `Retry-After`)

Examples

Basic

Prometheus servers push their metrics to port 9090. Every sender is let in, so use this only on a network where nobody else can reach the port...

- id: 1
  name: prometheus-remote-write
  description: Prometheus servers pushing their metrics over remote write
  type: prometheus_remote_write
  properties:
    address: "0.0.0.0"
    port: 9090
    authentication: none

TLS and a Token

The same over TLS, with senders that sign in with a token. To check the certificates of the senders (mTLS), name the CA that signed them; client_auth_required refuses a sender that presents none...

- id: 2
  name: prometheus-remote-write-secure
  description: Grafana Alloy and OpenTelemetry Collector pushing metrics over TLS with a token
  type: prometheus_remote_write
  properties:
    address: "0.0.0.0"
    port: 9090
    authentication: bearer
    token: "${PROMETHEUS_REMOTE_WRITE_TOKEN}"
    tls:
      status: true
      cert_name: "prometheus-remote-write.example.com.crt"
      key_name: "prometheus-remote-write.example.com.key"
      min_tls_version: "tls1.2"
      # client_ca_name: "senders-ca.crt"
      # client_auth_required: true

Synopsis​

Schema​

Configuration​

Device​

Listener​

Authentication​

Access List and Tenants​

TLS​

Ingest Buffer​

Pointing the Senders at the Device​

Signing In​

Answers​

The Headers and the Body​

Limits​

Records​

How the Device Finds the Type of a Metric​

What the Device Leaves Out​

Sending the Data On​

Changing the Configuration​

Clusters​

Console​

Troubleshooting​

Configuration Errors​

Errors When the Device Starts​

Log Lines While It Runs​

What the Sender Sees​

Examples​

Basic​

TLS and a Token​