Prometheus: Alertmanager – send alerts to a “/dev/null”

By | 03/26/2019
 

In addition to the Prometheus: Alertmanager’s alerts receivers and routing based on severity level and tags post.

Have an Alertmanager config with routes.

The task is – send all alerts from a Dev-environment into a “/dev/null”.

To do this – create an empty receiver:

...
receivers:

  - name: 'blackhole'

  - name: 'default'
    slack_configs:
      - send_resolved: true
        title_link: 'http://dev.monitor.example.world/prometheus/alerts'
        title: '{{ if eq .Status "firing" }}:confused:{{ else }}:dancing_panda:{{ end }} [{{ .Status | toUpper }}] {{ .CommonAnnotations.summary }}'
        text: "{{ range .Alerts }}*Priority*: `{{ .Labels.severity | toUpper }}`\nMonitoring host: {{ .Labels.monitor }}\n{{ .Annotations.description }}\n{{ end }}"
...

And in routes add new rules – check the env tag used in our monitoring with the match_re and send all matches to the receiver: blackhole:

...
  receiver: 'default'

  routes:

    # capture All INFO
    - match:
        severity: info
      receiver: default

      routes:
      # forward Dev INFO to the 'blackhole'
      - match_re:
          env: .*(-dev).*
        receiver: blackhole

    # capture All WARN to the 'warning' with P3
    - match:
        severity: warning
      receiver: warning

      routes:
      # forward Dev WARNING to the 'blackhole'
      - match_re:
          env: .*(-dev).*
        receiver: blackhole

      # forward Stage WARN to the 'default'
      - match_re:
          env: .*(-stage).*
        receiver: default

    # capture All CRIT to the 'critical' with P1
    - match:
        severity: critical
      receiver: critical

      routes:
      # forward Dev CRIT to the 'blackhole'
      - match_re:
          env: .*(-dev).*
        receiver: blackhole

      # forward Stage CRIT to the 'default'
      - match_re:
          env: .*(-stage).*
        receiver: default
...

Done.