In addition to the Prometheus: Alertmanager’s alerts receivers and routing based on severity level and tags post.
Have an Alertmanager config with routes.
The task is – send all alerts from a Dev-environment into a “/dev/null”.
To do this – create an empty receiver:
... receivers: - name: 'blackhole' - name: 'default' slack_configs: - send_resolved: true title_link: 'http://dev.monitor.example.world/prometheus/alerts' title: '{{ if eq .Status "firing" }}:confused:{{ else }}:dancing_panda:{{ end }} [{{ .Status | toUpper }}] {{ .CommonAnnotations.summary }}' text: "{{ range .Alerts }}*Priority*: `{{ .Labels.severity | toUpper }}`\nMonitoring host: {{ .Labels.monitor }}\n{{ .Annotations.description }}\n{{ end }}" ...
And in routes add new rules – check the env tag used in our monitoring with the match_re and send all matches to the receiver: blackhole
:
... receiver: 'default' routes: # capture All INFO - match: severity: info receiver: default routes: # forward Dev INFO to the 'blackhole' - match_re: env: .*(-dev).* receiver: blackhole # capture All WARN to the 'warning' with P3 - match: severity: warning receiver: warning routes: # forward Dev WARNING to the 'blackhole' - match_re: env: .*(-dev).* receiver: blackhole # forward Stage WARN to the 'default' - match_re: env: .*(-stage).* receiver: default # capture All CRIT to the 'critical' with P1 - match: severity: critical receiver: critical routes: # forward Dev CRIT to the 'blackhole' - match_re: env: .*(-dev).* receiver: blackhole # forward Stage CRIT to the 'default' - match_re: env: .*(-stage).* receiver: default ...
Done.