Alert Configuration for Self-Hosting

How alerts work in a self-hosted Databand instance.

This is a continuation of the Alerting page, relevant for self-hosted Databand.

How Alerts Work

When you configure alerts in the Databand Application's UI, the Databand web server sends information about the event that matches the alert condition to the Prometheus notification service where the alert rules are configured. Next, Prometheus sends information to the Alertmanager service which is responsible for defining alert receivers and sending alerts.

For more information about how Prometheus and Alertmanager work together, see Alerting overview.

To get started with alerts, you need to configure the following system components:

  • In the Databand Application UI, configure a sample alert with the condition that describes when you want the alert to be triggered.
  • In the configuration of the Prometheus web service, specify your alerting rules.
  • In the configuration of the Alertsmanager web service, specify where the alert needs to be sent (Databand application, Slack, email, etc.) and the users who will receive them.

Here is an example of a configuration specified in the Alertsmanager web service:

route:
  receiver: 'databand'
  routes:
    - receiver: 'databand'
      group_wait: 1s
      group_interval: 5s
      repeat_interval: 30s
      match:
        is_dbnd_internal: 'true'
      continue: false  # skip next receivers
    # - receiver: 'email'
    #   group_wait: 30s
    #   group_interval: 5m
    #   repeat_interval: 1h
    #   continue: true
receivers:
  - name: 'databand'
    webhook_configs:
      - url: 'http://webserver:8080/api/v1/webhooks/alertmanager'
        send_resolved: false
  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/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'
        username: 'DBND Alertmanager'
        channel: '#dbnd-alerts-testing'
        icon_url: https://avatars3.githubusercontent.com/u/3380462
        send_resolved: false
        title: '{{ template "dbnd_title" . }}'
        title_link: '{{ template "dbnd_titlelink" . }}'
        text: '{{ template "dbnd_slack_message" . }}'
  - name: 'email'
    email_configs:
      - to: '[email protected]'  # set receiver email
        from: '[email protected]'  # set verified Sender Identity
        smarthost: 'smtp.sendgrid.net:587'
        auth_username: 'apikey'
        auth_password: '<< API Token >>'
        send_resolved: false
        html: '{{ template "dbnd_email_message" . }}'
        headers:
          subject: '{{ template "dbnd_email_subject" . }}'
templates:
  - /etc/alertmanager/notifications.tmpl
  - /etc/alertmanager/email.tmpl

As you can see from the configuration example, in Alerts, you can also specify the conditions for sending alerts to the next alerts cycle.

πŸ“˜

Alert time range

Alerts will be triggered on the last 48 hours of metadata. For example, if you set alerts on failed runs, any failure within the last 48 hours will trigger alerts.

Alert cycles

When the Databand web server registers an event matching a specific alert condition specified in the Databand Application UI, an in-app alert is sent to the Databand web application.

The first cycle of alertsThe first cycle of alerts

The first cycle of alerts

If end users do not resolve or acknowledge the alert, the Databand web server initiates the second cycle of alerts, and email or slack notifications are sent to end users.

Second cycle of alertsSecond cycle of alerts

Second cycle of alerts

The second cycle of alerts is triggered within a time period specified in the repeat_interval field in the configuration of Alertsmanager. You can have multiple cycles of alerts configured.

Configure Alert Manager to Send Notifications to Slack

πŸ“˜

Note

If you have Slack configured as alerts receiver, in addition to already available details in the Slack notification message, it includes information about the user who triggered a pipeline run for which an alert was generated.

  1. Create a Slack incoming webhook.
    For information about how to create a webhook, see Slack online documentation.

  2. Point the Alert Manager to your webhook
    In the /configu/alertmanager/config.yml, uncomment the default Slack receiver, and specify the following fields:

  • slack_configs.api_url
  • slack_configs.channel
  • username.
  1. Uncomment and/or configure custom routes for your Slack notifications.

See Alert Manager online documentation for information about how to customize it.

Alert Manager is now ready to send notifications to your Slack.

🚧

Email alerting prerequisites

You must whitelist [email protected] email address. Email alerts from Databand will be sent from this address.


What’s Next
Did this page help you?