From 971f92e45d41c51f0fd464f2dd6b36c64f567d8a Mon Sep 17 00:00:00 2001 From: Pepe Cano <825430+ppcano@users.noreply.github.com> Date: Wed, 30 Apr 2025 11:15:51 +0200 Subject: [PATCH] docs(alerting): add recommendation to reduce duplicated `DatasourceError` alerts (#104679) docs(alerting): add recommendation to reduce `DatasourceError` alerts --- .../fundamentals/alert-rule-evaluation/state-and-health.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/sources/alerting/fundamentals/alert-rule-evaluation/state-and-health.md b/docs/sources/alerting/fundamentals/alert-rule-evaluation/state-and-health.md index b7987e29828..cf877958a02 100644 --- a/docs/sources/alerting/fundamentals/alert-rule-evaluation/state-and-health.md +++ b/docs/sources/alerting/fundamentals/alert-rule-evaluation/state-and-health.md @@ -39,6 +39,11 @@ refs: destination: /docs/grafana//alerting/fundamentals/notifications/ - pattern: /docs/grafana-cloud/ destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/ + notification-policies: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/fundamentals/notifications/notification-policies/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/notification-policies/ --- # State and health of alerts @@ -131,6 +136,8 @@ To minimize the number of **No Data** or **Error** state alerts received, try th 1. Change the default [evaluation time out](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#evaluation_timeout). The default is set at 30 seconds. To increase the default evaluation timeout, open a support ticket from the [Cloud Portal](https://grafana.com/docs/grafana-cloud/account-management/support/#grafana-cloud-support-options). Note that this should be a last resort, because it may affect the performance of all alert rules and cause missed evaluations if the timeout is too long. +1. To reduce multiple notifications from **Error** alerts, define a [notification policy](ref:notification-policies) to handle all related alerts with `alertname=DatasourceError`, and filter and group errors from the same data source using the `datasource_uid` label. + ### Keep last state The "Keep Last State" option helps mitigate temporary data source issues, preventing alerts from unintentionally firing, resolving, and re-firing.