Skip to main content

Amazon Managed Service for Prometheus

Amazon AWS Observability

Synopsis​

Creates an Amazon Managed Service for Prometheus target that sends metrics to a workspace with remote-write.

It is the Prometheus remote-write target with AWS's sign-in and limits built in. That page describes what the Prometheus targets have in common: what is sent, names and labels, counts over an interval, and the receiver's answers. This page covers only what differs.

Schema​

- name: <string>
  description: <string>
  type: amazonprometheus
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    region: <string>
    key: <string>
    secret: <string>
    session: <string>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration​

The following fields are used to define the target:

FieldRequiredDefaultDescription
nameYTarget name
descriptionN-Optional description
typeYMust be amazonprometheus
pipelinesN-Optional post-processor pipelines
statusNtrueEnable/disable the target

Connection​

FieldRequiredDefaultDescription
urlY-The workspace's remote-write endpoint, https://aps-workspaces.<region>.amazonaws.com/workspaces/<workspace-id>/api/v1/remote_write. The workspace's details in the AWS console show it. It must use https. In the console: Remote-write URL
regionN-The AWS region of the workspace. The target reads it from the URL. Set it when the URL does not name it, such as a private endpoint. In the console: Region

AWS Credentials​

FieldRequiredDefaultDescription
keyN-The access key ID. Set it with secret. In the console: Access key ID, when Credentials is Access key
secretN-The secret access key. Set it with key. In the console: Secret access key
sessionN-The session token of temporary credentials. It is used only with key and secret. In the console: Session token

key, secret and session can each be plain text, an environment variable or a secret. Leave out key and secret to sign with the credentials of the host (see Credentials).

Performance​

FieldRequiredDefaultDescription
timeoutN30As on the Prometheus remote-write page
batch_sizeN1000As on the Prometheus remote-write page. A request that would be over 1 MB is split (see Request Size)
max_retriesN3As on the Prometheus remote-write page
retry_delayN1As on the Prometheus remote-write page
disable_retriesNfalseAs on the Prometheus remote-write page

Scheduling​

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options​

The debug options work as on the OTLP page.

Settings That Do Not Apply​

The target signs each request itself, so it ignores username, password and token, and its signature replaces an Authorization header set under headers. It also ignores the settings listed on the Prometheus remote-write page.

The Console Form​

The form has one step, Amazon Managed Service for Prometheus settings:

  • Remote-write URL and Region (Optional).
  • Credentials, which is Host's role (the default) or Access key. Access key shows Access key ID and Secret access key, both required, and Session token (Optional). The secret access key has the Manual / Environment variable / Vault picker.
  • The Performance fields.

The Credentials choice belongs to the form, not to the YAML. Host's role sends no credentials, and Access key sends the key fields. The form has no TLS step, because AWS's endpoints use public certificates.

Details​

Sign-in​

The target signs each request with AWS Signature Version 4 for the service aps. The identity needs the aps:RemoteWrite permission on the workspace.

Credentials​

Two ways:

  • An access key: set key and secret, and session for temporary credentials. The target reads them when it starts, and it does not renew a session token when the token expires.
  • The credentials of the host: leave out key and secret. The target then finds credentials the same way the AWS SDK does. It checks the environment, then a shared profile, then the role of the host, such as an EC2 instance profile or an EKS service account role.

For temporary credentials, use the role of the host, or a shared profile that assumes a role, signs in with IAM Identity Center or runs a credential process. The AWS SDK renews these. A profile that holds temporary keys copied into it must be updated when they expire.

A key, a secret or a session token that is set but resolves to nothing stops the target from starting. So an empty secret never makes the target sign as the host by mistake.

Request Size​

AWS refuses a request over 1 MB. The target splits a request that would be larger, between two records. A single record over the limit by itself is dropped and counted as dropped, with a warning in the Director log.

The Workspace's Limits​

AWS limits each workspace, for example its active series and its samples per second. Over the ingestion rate, AWS refuses the whole request, and the target retries it. AWS also refuses samples older than one hour. The workspace's quotas are on the Service Quotas page of the AWS console.

Everything Else​

Everything else works as on the Prometheus remote-write page.

Examples​

The Role of the Host​

Sending with the role of the host...

targets:
  - name: amazon_prometheus
    type: amazonprometheus
    properties:
      url: "https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-12345678-abcd-1234-abcd-123456789012/api/v1/remote_write"

An Access Key​

An access key, with its secret from a secret store...

targets:
  - name: amazon_prometheus_key
    type: amazonprometheus
    properties:
      url: "https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-12345678-abcd-1234-abcd-123456789012/api/v1/remote_write"
      key: "${AWS_ACCESS_KEY_ID}"
      secret: "$secret{aws-prometheus-secret-key}"

A Private Endpoint​

A private endpoint whose address does not name the region...

targets:
  - name: amazon_prometheus_private
    type: amazonprometheus
    properties:
      url: "https://prometheus.internal.example.com/workspaces/ws-12345678-abcd-1234-abcd-123456789012/api/v1/remote_write"
      region: "eu-west-1"

Troubleshooting​

Configuration Errors​

Director refuses a configuration it cannot use, with one of these messages:

  • url is required for Amazon Managed Service for Prometheus
  • url for Amazon Managed Service for Prometheus must be an https URL with a host, such as https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-12345678/api/v1/remote_write
  • region is required for Amazon Managed Service for Prometheus: the url does not name its region, so set region, for example eu-west-1
  • set both key and secret, or neither
  • the secret of the Amazon Managed Service for Prometheus target resolved to nothing: check the environment variable or the secret it names: the setting is set, but its environment variable or secret is empty. The same message names the key or the session token.
  • failed to resolve the key of the Amazon Managed Service for Prometheus target: ...: the environment variable or the secret could not be read.
  • no AWS credentials to sign the requests with: set key and secret, or give the host an AWS role

Errors While It Runs​

  • [OTLP] sign-error ...: the target could not get AWS credentials to sign the request, for example because the host has no AWS role. The target retries, and the batches are delivered again once this is fixed.
  • status 403: the identity lacks aps:RemoteWrite, region does not match the workspace, or the credentials expired, such as a session token set by hand. The batches are delivered again once this is fixed.
  • status 404: the workspace ID or the path in url is wrong.
  • status 429: the workspace is over its ingestion rate. The target retries.
  • [OTLP] dropped a record of N bytes: Amazon Managed Service for Prometheus refuses a request over 1000000 bytes: a single record was too large to send.
  • The warnings of the Prometheus remote-write page apply too.