Repeat alerts within minutes using Grafana alerts

php editor Youzi will introduce to you today how to use the Grafana alarm function to repeat the alarm within a few minutes to ensure that important events are handled in a timely manner. Grafana is a popular open source data visualization and monitoring tool that provides rich dashboards and alerting capabilities. By setting alert rules and notification methods, we can detect abnormalities in the system in time and take appropriate measures. This article will introduce in detail how to use the Grafana alarm function, and share some practical tips and precautions so that you can easily deal with various monitoring scenarios.

Question content

Use Grafana 9.2.2 and VictoriaMetrics as data sources to send alerts when specific conditions are met. Use an external service to deliver alerts by configuring the API as a webhook touchpoint, through which the payload is sent and further processed for delivery on Slack.

Alarm evaluation behavior is set to - Evaluate every 1 hour for 0 seconds. You want the alert to be triggered immediately when the condition is met, and evaluated every 1 hour since that's how often new data points are.

Expected Behavior: An alert is sent every 24 hours when the condition is met.

Actual Behavior: Once the condition is met, the alert is triggered (as it should). However, the same alert is sent again within 5 minutes.

How to deal with this problem?

Tried options:

1. Notification Policy Timing - Tried grouping interval, repeat interval and wait time when using alertname and grafana-folder for grouping but to no avail. Also, I tried using alert_uid for grouping but this was not explained. Am I trying the wrong combination of times (combined with the alert evaluation behavior period).
2. After receiving the payload, do I need to send a confirmation to Grafana? If so, please share the method or link to any documentation you can find. I can't find anything that answers yes/no and how to answer it. I have isolated the issue to Grafana and the triggered API is being called twice.
3. There is an option to mute and/or mute the alarm. Is this the approach to follow here? If so, should the alarm be silenced for 24 hours after being triggered once (because you don't want to repeat the 24 hours)?

Thanks. Thank you so much.

Solution

The problem is that multiple instances of Grafana are running independently of each other. We have 2 Grafana pods running and they are both serving requests, so there is duplication. A future need to examine how to run Grafana in cluster mode.

The above is the detailed content of Repeat alerts within minutes using Grafana alerts. For more information, please follow other related articles on the PHP Chinese website!