Active alerts sending frequency via Alertmanager is configured via the repeat_interval in the /etc/alertmanager/config.yml file. We have this interval set to 15 minutes, and as result, we have notifications about alerts in our Slack each fifteen minutes. Still, some alerts are such a “known issues”, when we already started the investigation or fixing it, but… Read More »
We have three working environments – Dev, Stage, Production. Also, there are a bunch of alerts with different severities – info, warning и critical. For example: … – name: SSLexpiry.rules rules: – alert: SSLCertExpiring30days expr: probe_ssl_earliest_cert_expiry{job=”blackbox”} – time() < 86400 * 30 for: 10m labels: severity: info annotations: summary: “SSL certificate warning” description: “SSL certificate… Read More »
In addition to the Prometheus: Alertmanager’s alerts receivers and routing based on severity level and tags post. Have an Alertmanager config with routes. The task is – send all alerts from a Dev-environment into a “/dev/null”. To do this – create an empty receiver: … receivers: – name: ‘blackhole’ – name: ‘default’ slack_configs: – send_resolved:… Read More »