public/grafana
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Docs: Refactor alerting documentation (part 1) (#48664)

Browse Source 
* Initial commit

* Moved files, ad fixed broken relrefs.

* Fixed other broken relrefs

* More changes.

* Fixing broken relrefs

* More changes.

* Fixed last of the broken links

* More re-org.

* Added aliases and some weight adjustments

* More aliases.

* Fix fundamentals topic.

* Fixed remaining metadata issues

* Ran prettier
This commit is contained in:
achatterjee-grafana
2022-05-04 08:22:11 -04:00
committed by GitHub
parent f00ffb190c
commit 082cfbdb06
49 changed files with 148 additions and 120 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sources/alerting/_index.md
+21 -20
View File
@@ -1,35 +1,36 @@
+++
title = "Alerts"
weight = 110
weight = 455
aliases = ["/docs/grafana/latest/alerting/"]
+++
# Grafana alerts
Alerts allow you to learn about problems in your systems moments after they occur. Robust and actionable alerts help you identify and resolve issues quickly, minimizing disruption to your services.
Grafana 8.0 introduced new and improved alerting that centralizes alerting information in a single, searchable view. It allows you to:
Grafana alerts allow you to learn about problems in your systems moments after they occur. Robust and actionable alerts help you identify and resolve issues quickly, minimizing disruption to your services. It centralizes alerting information in a single, searchable view that allows you to:
- Create and manage Grafana alerts
- Create and manage Grafana Mimir and Loki managed alerts
- View alerting information from Prometheus and Alertmanager compatible data sources
Grafana alerting is enabled by default for new OSS installations. For older installations, it is still an [opt-in]({{< relref "./unified-alerting/opt-in.md" >}}) feature.
For new installations or existing installs without alerting configured, Grafana alerting is enabled by default.
| Release | Cloud | Enterprise | OSS |
| ------------------------ | ------------- | ------------- | -------------------------------- |
| Grafana 8.2 | On by default | Opt-in | Opt-in |
| Grafana 8.3 | On by default | Opt-in | On by default for new installs\* |
| Grafana 9.0 (unreleased) | On by default | On by default | On by default |
| Release | Cloud | Enterprise | OSS |
| ----------- | ------------- | ------------- | ------------- |
| Grafana 9.0 | On by default | On by default | On by default |
> **Note:** New installs include existing installs which do not have any alerts configured.
- For existing OSS installations with legacy dashboard alerting, you can [opt-in]({{< relref "./opt-in.md" >}}) to Grafana alerting.
- For Grafana Cloud instances using legacy cloud alerting, contact customer support to migrate to Grafana alerting.
Grafana alerting has four key components:
Before you begin, we recommend that you familiarize yourself with some of the [fundamental concepts]({{< relref "./fundamentals/_index.md" >}}) of Grafana alerting. Refer to [Fine-grained access control]({{< relref "../enterprise/access-control/_index.md" >}}) in Grafana Enterprise to learn more about controlling access to alerts using fine-grained permissions.
- Alerting rule - Evaluation criteria that determine whether an alert will fire. It consists of one or more queries and expressions, a condition, the frequency of evaluation, and optionally, the duration over which the condition is met.
- Contact point - Channel for sending notifications when the conditions of an alerting rule are met.
- Notification policy - Set of matching and grouping criteria used to determine where and how frequently to send notifications.
- Silences - Date and matching criteria used to silence notifications.
To learn more, see [What's New with Grafana alerting]({{< relref "../alerting/unified-alerting/difference-old-new.md" >}}).
For information on how to create and manage Grafana alerts and notifications, refer to [Overview of Grafana alerts]({{< relref "../alerting/unified-alerting/_index.md" >}}) and [Create and manage Grafana alerting rules]({{< relref "./unified-alerting/alerting-rules/_index.md" >}}).
- [What's new in Grafana alerting]({{< relref "./difference-old-new.md" >}})
- [Enable Grafana alerting in OSS]({{< relref "./opt-in.md" >}})
- [Migrating legacy alerts]({{< relref "./migrating-legacy-alerts.md" >}})
- [Create Grafana managed alerting rules]({{< relref "alerting-rules/create-grafana-managed-rule.md" >}})
- [Create Grafana Mimir or Loki managed alerting rules]({{< relref "alerting-rules/create-mimir-loki-managed-rule.md" >}})
- [View existing alerting rules and manage their current state]({{< relref "alerting-rules/rule-list.md" >}})
- [View the state and health of alerting rules]({{< relref "./fundamentals/state-and-health.md" >}})
- [View alert groupings]({{< relref "./alert-groups.md" >}})
- [Add or edit an alert contact point]({{< relref "./contact-points.md" >}})
- [Add or edit notification policies]({{< relref "./notifications/_index.md" >}})
- [Add or edit silences]({{< relref "./silences.md" >}})
docs/sources/alerting/unified-alerting/alert-groups.md → docs/sources/alerting/alert-groups.md
+1
View File
@@ -3,6 +3,7 @@ title = "Alert groups"
description = "Alert groups"
keywords = ["grafana", "alerting", "alerts", "groups"]
weight = 400
aliases = ["/docs/grafana/latest/alerting/unified-alerting/alert-groups/"]
+++
# Alert groups
docs/sources/alerting/alerting-limitations.md
+10
View File
@@ -0,0 +1,10 @@
+++
title = "Limitations"
weight = 552
aliases = ["/docs/grafana/latest/alerting/alerting-limitations/"]
+++
# Limitations
- Grafana alerting system can retrieve rules from all available Prometheus, Loki, and Alertmanager data sources. It might not be able to fetch alerting rules from all other supported data sources at this time.
- We aim to support the latest two minor versions of both Prometheus and Alertmanager. We cannot guarantee that older versions will work. As an example, if the current Prometheus version is `2.31.1`, we support >= `2.29.0`.
docs/sources/alerting/unified-alerting/alerting-rules/_index.md → docs/sources/alerting/alerting-rules/_index.md
+1 -1
View File
@@ -1,6 +1,6 @@
+++
title = "Create and manage rules"
aliases = ["/docs/grafana/latest/alerting/rules/"]
aliases = ["/docs/grafana/latest/alerting/rules/", "/docs/grafana/latest/alerting/unified-alerting/alerting-rules/"]
weight = 130
+++
docs/sources/alerting/unified-alerting/alerting-rules/alert-annotation-label.md → docs/sources/alerting/alerting-rules/alert-annotation-label.md
+1
View File
@@ -3,6 +3,7 @@ title = "Annotations and labels for alerting rules"
description = "Annotations and labels for alerting"
keywords = ["grafana", "alerting", "guide", "rules", "create"]
weight = 401
aliases = ["/docs/grafana/latest/alerting/unified-alerting/alerting-rules/alert-annotation-label/"]
+++
# Annotations and labels for alerting rules
docs/sources/alerting/unified-alerting/alerting-rules/create-grafana-managed-rule.md → docs/sources/alerting/alerting-rules/create-grafana-managed-rule.md
+3 -2
View File
@@ -3,6 +3,7 @@ title = "Create Grafana managed alert rule"
description = "Create Grafana managed alert rule"
keywords = ["grafana", "alerting", "guide", "rules", "create"]
weight = 400
aliases = ["/docs/grafana/latest/alerting/unified-alerting/alerting-rules/create-grafana-managed-rule/"]
+++
# Create a Grafana managed alerting rule
@@ -20,7 +21,7 @@ Grafana allows you to create alerting rules that query one or more data sources,
1. In Step 2, add queries and expressions to evaluate.
- Keep the default name or hover over and click the edit icon to change the name.
- For queries, select a data source from the drop-down.
- Add one or more [queries]({{< relref "../../../panels/query-a-data-source/add-a-query.md" >}}) or [expressions]({{< relref "../../../panels/query-a-data-source/use-expressions-to-manipulate-data/about-expressions.md" >}}).
- Add one or more [queries]({{< relref "../../panels/query-a-data-source/add-a-query.md" >}}) or [expressions]({{< relref "../../panels/query-a-data-source/use-expressions-to-manipulate-data/about-expressions.md" >}}).
- For each expression, select either **Classic condition** to create a single alert rule, or choose from **Math**, **Reduce**, **Resample** options to generate separate alert for each series. For details on these options, see [Single and multi dimensional rule](#single-and-multi-dimensional-rule).
- Click **Run queries** to verify that the query is successful.
1. In Step 3, add conditions.
@@ -57,7 +58,7 @@ To generate a separate alert for each series, create a multi-dimensional rule. U
#### Rule with classic condition
For more information, see [expressions documentation]({{< relref "../../../panels/query-a-data-source/use-expressions-to-manipulate-data/about-expressions.md" >}}).
For more information, see [expressions documentation]({{< relref "../../panels/query-a-data-source/use-expressions-to-manipulate-data/about-expressions.md" >}}).
### No data and error handling
docs/sources/alerting/unified-alerting/alerting-rules/create-mimir-loki-managed-recording-rule.md → docs/sources/alerting/alerting-rules/create-mimir-loki-managed-recording-rule.md
+2
View File
@@ -3,6 +3,8 @@ title = "Create Grafana Mimir or Loki managed recording rule"
description = "Create Grafana Mimir or Loki managed recording rule"
keywords = ["grafana", "alerting", "guide", "rules", "recording rules", "create"]
weight = 400
aliases = ["/docs/grafana/latest/alerting/unified-alerting/alerting-rules/create-mimir-loki-managed-recording-rule/", "/docs/grafana/latest/alerting/unified-alerting/alerting-rules/create-cortex-loki-managed-recording-rule/"]
+++
# Create a Grafana Mimir or Loki managed recording rule
docs/sources/alerting/unified-alerting/alerting-rules/create-mimir-loki-managed-rule.md → docs/sources/alerting/alerting-rules/create-mimir-loki-managed-rule.md
+1 -1
View File
@@ -2,7 +2,7 @@
title = "Create Grafana Mimir or Loki managed alert rule"
description = "Create Grafana Mimir or Loki managed alerting rule"
keywords = ["grafana", "alerting", "guide", "rules", "create"]
aliases = ["/docs/grafana/latest/alerting/unified-alerting/alerting-rules/create-cortex-loki-managed-recording-rule/"]
aliases = ["/docs/grafana/latest/alerting/unified-alerting/alerting-rules/create-cortex-loki-managed-recording-rule/", "/docs/grafana/latest/alerting/unified-alerting/alerting-rules/create-mimir-loki-managed-recording-rule/"]
weight = 400
+++
docs/sources/alerting/unified-alerting/alerting-rules/edit-mimir-loki-namespace-group.md → docs/sources/alerting/alerting-rules/edit-mimir-loki-namespace-group.md
+1
View File
@@ -3,6 +3,7 @@ title = "Grafana Mimir or Loki rule groups and namespaces"
description = "Edit Grafana Mimir or Loki rule groups and namespaces"
keywords = ["grafana", "alerting", "guide", "group", "namespace", "grafana mimir", "loki"]
weight = 405
aliases = ["/docs/grafana/latest/alerting/unified-alerting/alerting-rules/edit-mimir-loki-namespace-group/", "/docs/grafana/latest/alerting/unified-alerting/alerting-rules/edit-cortex-loki-namespace-group/"]
+++
# Grafana Mimir or Loki rule groups and namespaces
docs/sources/alerting/unified-alerting/alerting-rules/rule-list.md → docs/sources/alerting/alerting-rules/rule-list.md
+1
View File
@@ -3,6 +3,7 @@ title = "Manage alerting rules"
description = "Manage alerting rules"
keywords = ["grafana", "alerting", "guide", "rules", "view"]
weight = 402
aliases = ["/docs/grafana/latest/alerting/unified-alerting/alerting-rules/rule-list/"]
+++
# Manage alerting rules
docs/sources/alerting/unified-alerting/contact-points.md → docs/sources/alerting/contact-points.md
+3 -2
View File
@@ -3,13 +3,14 @@ title = "Contact points"
description = "Create or edit contact point"
keywords = ["grafana", "alerting", "guide", "contact point", "notification channel", "create"]
weight = 430
aliases = ["/docs/grafana/latest/alerting/unified-alerting/contact-points/"]
+++
# Contact points
Use contact points to define how your contacts are notified when an alert fires. A contact point can have one or more contact point types, for example, email, slack, webhook, and so on. When an alert fires, a notification is sent to all contact point types listed for a contact point. Optionally, use [message templates]({{< relref "./message-templating/_index.md" >}}) to customize notification messages for the contact point types.
You can configure Grafana managed contact points as well as contact points for an [external Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}). For more information, see [Alertmanager]({{< relref "./fundamentals/alertmanager.md" >}}).
You can configure Grafana managed contact points as well as contact points for an [external Alertmanager data source]({{< relref "../datasources/alertmanager.md" >}}). For more information, see [Alertmanager]({{< relref "./fundamentals/alertmanager.md" >}}).
## Add a contact point
@@ -60,7 +61,7 @@ To edit global configuration options for an external Alertmanager, like SMTP ser
1. Add global configuration settings.
1. Click **Save global config** to save your changes.
> **Note** This option is available only for external Alertmanagers. You can configure some global options for Grafana contact types, like email settings, via [Grafana configuration]({{< relref "../../administration/configuration.md" >}}).
> **Note** This option is available only for external Alertmanagers. You can configure some global options for Grafana contact types, like email settings, via [Grafana configuration]({{< relref "../administration/configuration.md" >}}).
## List of notifiers supported by Grafana
docs/sources/alerting/unified-alerting/difference-old-new.md → docs/sources/alerting/difference-old-new.md
+2 -1
View File
@@ -2,7 +2,8 @@
title = "What's new in Grafana alerting"
description = "What's New with Grafana alerts"
keywords = ["grafana", "alerting", "guide"]
weight = 114
aliases = ["/docs/grafana/latest/alerting/unified-alerting/difference-old-new/"]
weight = 108
+++
# What's new in Grafana alerting
docs/sources/alerting/unified-alerting/fundamentals/_index.md → docs/sources/alerting/fundamentals/_index.md
+2 -2
View File
@@ -1,7 +1,7 @@
+++
title = "Alerting fundamentals"
aliases = ["/docs/grafana/latest/alerting/metrics/"]
weight = 120
weight = 110
aliases = ["/docs/grafana/latest/alerting/metrics/", "/docs/grafana/latest/alerting/unified-alerting/fundamentals/"]
+++
# Alerting fundamentals
docs/sources/alerting/unified-alerting/fundamentals/alertmanager.md → docs/sources/alerting/fundamentals/alertmanager.md
+2 -2
View File
@@ -1,6 +1,6 @@
+++
title = "Alertmanager"
aliases = ["/docs/grafana/latest/alerting/metrics/"]
aliases = ["/docs/grafana/latest/alerting/metrics/", "/docs/grafana/latest/alerting/unified-alerting/fundamentals/alertmanager/"]
weight = 116
+++
@@ -12,7 +12,7 @@ Grafana includes built-in support for Prometheus Alertmanager. By default, notif
> **Note:** Before v8.2, the configuration of the embedded Alertmanager was shared across organizations. If you are on an older Grafana version, we recommend that you use Grafana alerts only if you have one organization. Otherwise, your contact points are visible to all organizations.
Grafana alerting added support for external Alertmanager configuration. When you add an [Alertmanager data source]({{< relref "../../../datasources/alertmanager.md" >}}), the Alertmanager drop-down shows a list of available external Alertmanager data sources. Select a data source to create and manage alerting for standalone Grafana Mimir or Loki data sources.
Grafana alerting added support for external Alertmanager configuration. When you add an [Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}), the Alertmanager drop-down shows a list of available external Alertmanager data sources. Select a data source to create and manage alerting for standalone Grafana Mimir or Loki data sources.
{{< figure max-width="40%" src="/static/img/docs/alerting/unified/contact-points-select-am-8-0.gif" max-width="250px" caption="Select Alertmanager" >}}
docs/sources/alerting/unified-alerting/fundamentals/evaluate-grafana-alerts.md → docs/sources/alerting/fundamentals/evaluate-grafana-alerts.md
+8 -4
View File
@@ -1,6 +1,6 @@
+++
title = "Alerting on numeric data"
aliases = ["/docs/grafana/latest/alerting/metrics/"]
aliases = ["/docs/grafana/latest/alerting/metrics/", "/docs/grafana/latest/alerting/unified-alerting/fundamentals/evaluate-grafana-alerts/"]
weight = 116
+++
@@ -8,8 +8,12 @@ weight = 116
This topic describes how Grafana managed alerts are evaluated by the backend engine as well as how Grafana handles alerting on numeric rather than time series data.
- [Alert evaluation](#alert-evaluation)
- [Alerting on numeric data](#alerting-on-numeric-data)
- [Alert evaluation](#alert-evaluation)
- [Metrics from the alerting engine](#metrics-from-the-alerting-engine)
- [Alerting on numeric data](#alerting-on-numeric-data-1)
- [Tabular Data](#tabular-data)
- [Example](#example)
## Alert evaluation
@@ -17,11 +21,11 @@ Grafana managed alerts query the following backend data sources that have alerti
- built-in data sources or those developed and maintained by Grafana: `Graphite`, `Prometheus`, `Loki`, `InfluxDB`, `Elasticsearch`,
`Google Cloud Monitoring`, `Cloudwatch`, `Azure Monitor`, `MySQL`, `PostgreSQL`, `MSSQL`, `OpenTSDB`, `Oracle`, and `Azure Monitor`
- community developed backend data sources with alerting enabled (`backend` and `alerting` properties are set in the [plugin.json]({{< relref "../../../developers/plugins/metadata.md" >}}))
- community developed backend data sources with alerting enabled (`backend` and `alerting` properties are set in the [plugin.json]({{< relref "../../developers/plugins/metadata.md" >}}))
### Metrics from the alerting engine
The alerting engine publishes some internal metrics about itself. You can read more about how Grafana publishes [internal metrics]({{< relref "../../../administration/view-server/internal-metrics.md" >}}). See also, [View alert rules and their current state]({{< relref "../alerting-rules/rule-list.md" >}}).
The alerting engine publishes some internal metrics about itself. You can read more about how Grafana publishes [internal metrics]({{< relref "../../administration/view-server/internal-metrics.md" >}}). See also, [View alert rules and their current state]({{< relref "../alerting-rules/rule-list.md" >}}).
| Metric Name | Type | Description |
| ------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------- |
docs/sources/alerting/unified-alerting/fundamentals/state-and-health.md → docs/sources/alerting/fundamentals/state-and-health.md
View File
docs/sources/alerting/unified-alerting/high-availability/_index.md → docs/sources/alerting/high-availability/_index.md
+2 -1
View File
@@ -1,8 +1,9 @@
+++
title = " About alerting high availability"
title = " Alerting high availability"
description = "High availability"
keywords = ["grafana", "alerting", "tutorials", "ha", "high availability"]
aliases = ["/docs/grafana/latest/alerting/unified-alerting/high-availability/"]
weight = 450
+++
docs/sources/alerting/unified-alerting/high-availability/enable-alerting-ha.md → docs/sources/alerting/high-availability/enable-alerting-ha.md
View File
docs/sources/alerting/unified-alerting/message-templating/_index.md → docs/sources/alerting/message-templating/_index.md
+1 -1
View File
@@ -1,8 +1,8 @@
+++
title = "Message templating"
description = "Message templating"
aliases = ["/docs/grafana/latest/alerting/message-templating/"]
keywords = ["grafana", "alerting", "guide", "contact point", "templating"]
aliases = ["/docs/grafana/latest/alerting/message-templating/", "/docs/grafana/latest/alerting/unified-alerting/message-templating/"]
weight = 440
+++
docs/sources/alerting/unified-alerting/message-templating/template-data.md → docs/sources/alerting/message-templating/template-data.md
+1
View File
@@ -1,6 +1,7 @@
+++
title = "Template data"
keywords = ["grafana", "alerting", "guide", "contact point", "templating"]
aliases = ["/docs/grafana/latest/alerting/unified-alerting/message-templating/template-data/"]
+++
# Template data
docs/sources/alerting/unified-alerting/message-templating/template-functions.md → docs/sources/alerting/message-templating/template-functions.md
+1
View File
@@ -1,6 +1,7 @@
+++
title = "Template functions"
keywords = ["grafana", "alerting", "guide", "contact point", "templating"]
aliases = ["/docs/grafana/latest/alerting/unified-alerting/message-templating/template-functions/"]
+++
# Template Functions
docs/sources/alerting/unified-alerting/opt-in.md → docs/sources/alerting/migrating-legacy-alerts.md
+7 -29
View File
@@ -1,35 +1,13 @@
+++
title = "Opt-in to Grafana alerting"
title = "Migrating legacy alerts"
description = "Enable Grafana alerts"
weight = 115
weight = 114
aliases = ["/docs/grafana/latest/alerting/unified-alerting/opt-in/"]
+++
# Opt-in to Grafana alerting
# Migrating legacy alerts to Grafana alerting
Grafana alerting is enabled by default for new Cloud and OSS installations.
**Note:** If you are an existing Grafana Cloud user and want to explore unified alerting, contact Grafana Support. They will enable unified alerting for your Cloud stack.
For older OSS installations that use legacy dashboard alerts, unified alerting is still an opt-in feature. This topic describes how to opt-in to Grafana alerting if you have an existing Grafana installation and the rules and restrictions that govern the migration of existing dashboard alerts to the new alerting system. You can [disable Grafana alerts]({{< relref "./opt-in.md#disable-grafana-alerts" >}}) and use the legacy dashboard alerting if needed.
Before you begin, we recommend that you backup Grafana's database. If you are using PostgreSQL as the backend database, then the minimum required version is 9.5.
## Enable Grafana alerting
To enable Grafana alerts:
1. In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the [unified alerts]({{< relref "../../administration/configuration.md#unified_alerting" >}}) section.
1. Set the `enabled` property to `true`.
1. Next, for [legacy dashboard alerting]({{< relref "../../administration/configuration.md#alerting" >}}), set the `enabled` flag to `false`.
1. Restart Grafana for the configuration changes to take effect.
> **Note:** The `ngalert` toggle previously used to enable or disable Grafana alerting is no longer available.
Before v8.2, notification logs and silences were stored on a disk. If you did not use persistent disks, you would have lost any configured silences and logs on a restart, resulting in unwanted or duplicate notifications. We no longer require the use of a persistent disk. Instead, the notification logs and silences are stored regularly (every 15 minutes). If you used the file-based approach, Grafana reads the existing file and persists it eventually.
## Migrating legacy alerts to Grafana alerting system
When Grafana alerting is enabled or Grafana is upgraded to version 8.3, existing legacy dashboard alerts migrate in a format compatible with the Grafana alerting. In the Alerting page of your Grafana instance, you can view the migrated alerts alongside new alerts.
When Grafana alerting is enabled or Grafana is upgraded to the latest version, existing legacy dashboard alerts migrate in a format compatible with the Grafana alerting. In the Alerting page of your Grafana instance, you can view the migrated alerts alongside new alerts.
Read and write access to legacy dashboard alerts and Grafana alerts are governed by the permissions of the folders storing them. During migration, legacy dashboard alert permissions are matched to the new rules permissions as follows:
@@ -52,9 +30,9 @@ Grafana alerting system can retrieve rules from all available Prometheus, Loki,
To disable Grafana alerts and enable legacy dashboard alerts:
1. In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the [Grafana alerting]({{< relref "../../administration/configuration.md#unified_alerting" >}}) section.
1. In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the [Grafana alerting]({{< relref "../administration/configuration.md#unified_alerting" >}}) section.
1. Set the `enabled` property to `false`.
1. For [legacy dashboard alerting]({{< relref "../../administration/configuration.md#alerting" >}}), set the `enabled` flag to `true`.
1. For [legacy dashboard alerting]({{< relref "../administration/configuration.md#alerting" >}}), set the `enabled` flag to `true`.
1. Restart Grafana for the configuration changes to take effect.
> **Note:** Switching from one flavor of alerting to another can result in data loss. This is applicable to the fresh installation as well as upgraded setups.
docs/sources/alerting/unified-alerting/notifications/_index.md → docs/sources/alerting/notifications/_index.md
+2 -1
View File
@@ -3,13 +3,14 @@ title = "Notification policies"
description = "Notification policies"
keywords = ["grafana", "alerting", "guide", "notification policies", "routes"]
weight = 450
aliases = ["/docs/grafana/latest/alerting/unified-alerting/notifications/"]
+++
# Notification policies
Notification policies determine how alerts are routed to contact points. Policies have a tree structure, where each policy can have one or more child policies. Each policy, except for the root policy, can also match specific alert labels. Each alert is evaluated by the root policy and subsequently by each child policy. If you enable the `Continue matching subsequent sibling nodes` option is enabled for a specific policy, then evaluation continues even after one or more matches. A parent policy’s configuration settings and contact point information govern the behavior of an alert that does not match any of the child policies. A root policy governs any alert that does not match a specific policy.
You can configure Grafana managed notification policies as well as notification policies for an [external Alertmanager data source]({{< relref "../../../datasources/alertmanager.md" >}}). For more information, see [Alertmanager]({{< relref "../fundamentals/alertmanager.md" >}}).
You can configure Grafana managed notification policies as well as notification policies for an [external Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}). For more information, see [Alertmanager]({{< relref "../fundamentals/alertmanager.md" >}}).
## Grouping
docs/sources/alerting/unified-alerting/notifications/mute-timings.md → docs/sources/alerting/notifications/mute-timings.md
+2 -1
View File
@@ -3,6 +3,7 @@ title = "Mute timings"
description = "Mute timings"
keywords = ["grafana", "alerting", "guide", "mute", "mute timings", "mute time interval"]
weight = 450
aliases = ["/docs/grafana/latest/alerting/unified-alerting/notifications/mute-timings/"]
+++
# Mute timings
@@ -11,7 +12,7 @@ A mute timing is a recurring interval of time when no new notifications for a po
Similar to silences, mute timings do not prevent alert rules from being evaluated, nor do they stop alert instances from being shown in the user interface. They only prevent notifications from being created.
You can configure Grafana managed mute timings as well as mute timings for an [external Alertmanager data source]({{< relref "../../../datasources/alertmanager.md" >}}). For more information, see [Alertmanager]({{< relref "../fundamentals/alertmanager.md" >}}).
You can configure Grafana managed mute timings as well as mute timings for an [external Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}). For more information, see [Alertmanager documentation]({{< relref "../fundamentals/alertmanager.md" >}}).
## Mute timings vs silences
docs/sources/alerting/old-alerting/_index.md
-26
View File
@@ -1,26 +0,0 @@
+++
title = "Legacy Grafana Alerts"
weight = 114
+++
# Legacy Grafana alerts
Grafana alerting is enabled by default for new OSS installations. For older installations, it is still an [opt-in]({{< relref "../unified-alerting/opt-in.md" >}}) feature.
> **Note**: Legacy dashboard alerts are deprecated and will be removed in Grafana 9. We encourage you to migrate to [Grafana alerting]({{< relref "../unified-alerting/_index.md" >}}) for all existing installations.
Legacy dashboard alerts have two main components:
- Alert rule - When the alert is triggered. Alert rules are defined by one or more conditions that are regularly evaluated by Grafana.
- Notification channel - How the alert is delivered. When the conditions of an alert rule are met, the Grafana notifies the channels configured for that alert.
## Alert tasks
You can perform the following tasks for alerts:
- [Create an alert rule]({{< relref "create-alerts.md" >}})
- [View existing alert rules and their current state]({{< relref "view-alerts.md" >}})
- [Test alert rules and troubleshoot]({{< relref "troubleshoot-alerts.md" >}})
- [Add or edit an alert contact point]({{< relref "notifications.md" >}})
{{< docs/shared "alerts/grafana-managed-alerts.md" >}}
docs/sources/alerting/old-alerting/add-notification-template.md
-26
View File
@@ -1,26 +0,0 @@
+++
title = "Alert notification templating"
keywords = ["grafana", "documentation", "alerting", "alerts", "notification", "templating"]
weight = 110
aliases = ["/docs/grafana/latest/alerting/add-notification-template/"]
+++
# Alert notification templating
You can provide detailed information to alert notification recipients by injecting alert query data into an alert notification. This topic explains how you can use alert query labels in alert notifications.
You can use labels generated during an alerting query evaluation to create alert notification messages. For multiple unique values for the same label, the values are comma-separated.
When an alert fires, the alerting data series indicates the violation. For resolved alerts, all data series are included in the resolved notification.
This topic explains how you can use alert query labels in alert notifications.
## Adding alert label data into your alert notification
1. Navigate to the panel you want to add or edit an alert rule for.
1. Click on the panel title, and then click **Edit**.
1. On the Alert tab, click **Create Alert**. If an alert already exists for this panel, then you can edit the alert directly.
1. Refer to the alert query labels in the alert rule name and/or alert notification message field by using the `${Label}` syntax.
1. Click **Save** in the upper right corner to save the alert rule and the dashboard.
![Alerting notification template](/static/img/docs/alerting/alert-notification-template-7-4.png)
docs/sources/alerting/old-alerting/create-alerts.md
-126
View File
@@ -1,126 +0,0 @@
+++
title = "Create alerts"
description = "Configure alert rules"
keywords = ["grafana", "alerting", "guide", "rules"]
weight = 200
aliases = ["/docs/grafana/latest/alerting/create-alerts/"]
+++
# Create alerts
Grafana alerting allows you to attach rules to your dashboard panels. When you save the dashboard, Grafana extracts the alert rules into a separate alert rule storage and schedules them for evaluation.
![Alerting overview](/static/img/docs/alerting/drag_handles_gif.gif)
In the Alert tab of the graph panel you can configure how often the alert rule should be evaluated and the conditions that need to be met for the alert to change state and trigger its [notifications]({{< relref "notifications.md" >}}).
Currently only the graph panel supports alert rules.
## Add or edit an alert rule
1. Navigate to the panel you want to add or edit an alert rule for, click the title, and then click **Edit**.
1. On the Alert tab, click **Create Alert**. If an alert already exists for this panel, then you can just edit the fields on the Alert tab.
1. Fill out the fields. Descriptions are listed below in [Alert rule fields](#alert-rule-fields).
1. When you have finished writing your rule, click **Save** in the upper right corner to save alert rule and the dashboard.
1. (Optional but recommended) Click **Test rule** to make sure the rule returns the results you expect.
## Delete an alert
To delete an alert, scroll to the bottom of the alert and then click **Delete**.
## Alert rule fields
This section describes the fields you fill out to create an alert.
### Rule
- **Name -** Enter a descriptive name. The name will be displayed in the Alert Rules list. This field supports [templating]({{< relref "./add-notification-template.md" >}}).
- **Evaluate every -** Specify how often the scheduler should evaluate the alert rule. This is referred to as the _evaluation interval_.
- **For -** Specify how long the query needs to violate the configured thresholds before the alert notification triggers.
You can set a minimum evaluation interval in the `alerting.min_interval_seconds` configuration field, to set a minimum time between evaluations. Refer to [Configuration]({{< relref "../../administration/configuration.md" >}}#min-interval-seconds) for more information.
> **Caution:** Do not use `For` with the `If no data or all values are null` setting set to `No Data`. The triggering of `No Data` will trigger instantly and not take `For` into consideration. This may also result in that an OK notification not being sent if alert transitions from `No Data -> Pending -> OK`.
If an alert rule has a configured `For` and the query violates the configured threshold, then it will first go from `OK` to `Pending`. Going from `OK` to `Pending` Grafana will not send any notifications. Once the alert rule has been firing for more than `For` duration, it will change to `Alerting` and send alert notifications.
Typically, it's always a good idea to use this setting since it's often worse to get false positive than wait a few minutes before the alert notification triggers. Looking at the `Alert list` or `Alert list panels` you will be able to see alerts in pending state.
Below you can see an example timeline of an alert using the `For` setting. At ~16:04 the alert state changes to `Pending` and after 4 minutes it changes to `Alerting` which is when alert notifications are sent. Once the series falls back to normal the alert rule goes back to `OK`.
{{< figure class="float-right" src="/static/img/docs/v54/alerting-for-dark-theme.png" caption="Alerting For" >}}
{{< figure class="float-right" max-width="40%" src="/static/img/docs/v4/alerting_conditions.png" caption="Alerting Conditions" >}}
### Conditions
Currently the only condition type that exists is a `Query` condition that allows you to
specify a query letter, time range and an aggregation function.
#### Query condition example
```sql
avg() OF query(A, 15m, now) IS BELOW 14
```
- `avg()` Controls how the values for **each** series should be reduced to a value that can be compared against the threshold. Click on the function to change it to another aggregation function.
- `query(A, 15m, now)` The letter defines what query to execute from the **Metrics** tab. The second two parameters define the time range, `15m, now` means 15 minutes ago to now. You can also do `10m, now-2m` to define a time range that will be 10 minutes ago to 2 minutes ago. This is useful if you want to ignore the last 2 minutes of data.
- `IS BELOW 14` Defines the type of threshold and the threshold value. You can click on `IS BELOW` to change the type of threshold.
The query used in an alert rule cannot contain any template variables. Currently we only support `AND` and `OR` operators between conditions and they are executed serially.
For example, we have 3 conditions in the following order:
_condition:A(evaluates to: TRUE) OR condition:B(evaluates to: FALSE) AND condition:C(evaluates to: TRUE)_
so the result will be calculated as ((TRUE OR FALSE) AND TRUE) = TRUE.
We plan to add other condition types in the future, like `Other Alert`, where you can include the state of another alert in your conditions, and `Time Of Day`.
#### Multiple Series
If a query returns multiple series, then the aggregation function and threshold check will be evaluated for each series. What Grafana does not do currently is track alert rule state **per series**. This has implications that are detailed in the scenario below.
- Alert condition with query that returns 2 series: **server1** and **server2**
- **server1** series causes the alert rule to fire and switch to state `Alerting`
- Notifications are sent out with message: _load peaking (server1)_
- In a subsequent evaluation of the same alert rule, the **server2** series also causes the alert rule to fire
- No new notifications are sent as the alert rule is already in state `Alerting`.
So, as you can see from the above scenario Grafana will not send out notifications when other series cause the alert to fire if the rule already is in state `Alerting`. To improve support for queries that return multiple series we plan to track state **per series** in a future release.
> Starting with Grafana v5.3 you can configure reminders to be sent for triggered alerts. This will send additional notifications
> when an alert continues to fire. If other series (like server2 in the example above) also cause the alert rule to fire they will be included in the reminder notification. Depending on what notification channel you're using you may be able to take advantage of this feature for identifying new/existing series causing alert to fire.
### No Data & Error Handling
Below are conditions you can configure how the rule evaluation engine should handle queries that return no data or only null values.
| No Data Option | Description |
| --------------- | ------------------------------------------------------------------------------------------ |
| No Data | Set alert rule state to `NoData` |
| Alerting | Set alert rule state to `Alerting` |
| Keep Last State | Keep the current alert rule state, whatever it is. |
| Ok | Not sure why you would want to send yourself an alert when things are okay, but you could. |
### Execution errors or timeouts
Tell Grafana how to handle execution or timeout errors.
| Error or timeout option | Description |
| ----------------------- | -------------------------------------------------- |
| Alerting | Set alert rule state to `Alerting` |
| Keep Last State | Keep the current alert rule state, whatever it is. |
If you have an unreliable time series store from which queries sometime timeout or fail randomly you can set this option to `Keep Last State` in order to basically ignore them.
## Notifications
In alert tab you can also specify alert rule notifications along with a detailed message about the alert rule. The message can contain anything, information about how you might solve the issue, link to runbook, and so on.
The actual notifications are configured and shared between multiple alerts. Read
[Alert notifications]({{< relref "notifications.md" >}}) for information on how to configure and set up notifications.
- **Send to -** Select an alert notification channel if you have one set up.
- **Message -** Enter a text message to be sent on the notification channel. Some alert notifiers support transforming the text to HTML or other rich formats. This field supports [templating]({{< relref "./add-notification-template.md" >}}).
- **Tags -** Specify a list of tags (key/value) to be included in the notification. It is only supported by [some notifiers]({{< relref "notifications/#all-supported-notifiers" >}}).
## Alert state history and annotations
Alert state changes are recorded in the internal annotation table in Grafana's database. The state changes are visualized as annotations in the alert rule's graph panel. You can also go into the `State history` submenu in the alert tab to view and clear state history.
docs/sources/alerting/old-alerting/notifications.md
-282
View File
@@ -1,282 +0,0 @@
+++
title = "Alert notifications"
description = "Alerting notifications guide"
keywords = ["Grafana", "alerting", "guide", "notifications"]
weight = 100
aliases = ["/docs/grafana/latest/alerting/notifications/"]
+++
# Alert notifications
When an alert changes state, it sends out notifications. Each alert rule can have
multiple notifications. In order to add a notification to an alert rule you first need
to add and configure a `notification` channel (can be email, PagerDuty, or other integration).
This is done from the Notification channels page.
> **Note:** Alerting is only available in Grafana v4.0 and above.
## Add a notification channel
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Notification channels**.
1. Click **Add channel**.
1. Fill out the fields or select options described below.
## New notification channel fields
### Default (send on all alerts)
- **Name -** Enter a name for this channel. It will be displayed when users add notifications to alert rules.
- **Type -** Select the channel type. Refer to the [List of supported notifiers](#list-of-supported-notifiers) for details.
- **Default (send on all alerts) -** When selected, this option sends a notification on this channel for all alert rules.
- **Include Image -** See [Enable images in notifications](#enable-images-in-notifications-external-image-store) for details.
- **Disable Resolve Message -** When selected, this option disables the resolve message [OK] that is sent when the alerting state returns to false.
- **Send reminders -** When this option is checked additional notifications (reminders) will be sent for triggered alerts. You can specify how often reminders should be sent using number of seconds (s), minutes (m) or hours (h), for example `30s`, `3m`, `5m` or `1h`.
**Important:** Alert reminders are sent after rules are evaluated. Therefore a reminder can never be sent more frequently than a configured alert rule evaluation interval.
These examples show how often and when reminders are sent for a triggered alert.
| Alert rule evaluation interval | Send reminders every | Reminder sent every (after last alert notification) |
| ------------------------------ | -------------------- | --------------------------------------------------- |
| `30s` | `15s` | ~30 seconds |
| `1m` | `5m` | ~5 minutes |
| `5m` | `15m` | ~15 minutes |
| `6m` | `20m` | ~24 minutes |
| `1h` | `15m` | ~1 hour |
| `1h` | `2h` | ~2 hours |
<div class="clearfix"></div>
## List of supported notifiers
| Name | Type | Supports images | Supports alert rule tags |
| --------------------------------------------- | ------------------------- | ------------------ | ------------------------ |
| [DingDing](#dingdingdingtalk) | `dingding` | yes, external only | no |
| [Discord](#discord) | `discord` | yes | no |
| [Email](#email) | `email` | yes | no |
| [Google Hangouts Chat](#google-hangouts-chat) | `googlechat` | yes, external only | no |
| Hipchat | `hipchat` | yes, external only | no |
| [Kafka](#kafka) | `kafka` | yes, external only | no |
| Line | `line` | yes, external only | no |
| Microsoft Teams | `teams` | yes, external only | no |
| [Opsgenie](#opsgenie) | `opsgenie` | yes, external only | yes |
| [Pagerduty](#pagerduty) | `pagerduty` | yes, external only | yes |
| Prometheus Alertmanager | `prometheus-alertmanager` | yes, external only | yes |
| [Pushover](#pushover) | `pushover` | yes | no |
| Sensu | `sensu` | yes, external only | no |
| [Sensu Go](#sensu-go) | `sensugo` | yes, external only | no |
| [Slack](#slack) | `slack` | yes | no |
| Telegram | `telegram` | yes | no |
| Threema | `threema` | yes, external only | no |
| VictorOps | `victorops` | yes, external only | yes |
| [Webhook](#webhook) | `webhook` | yes, external only | yes |
### Email
To enable email notifications you have to set up [SMTP settings]({{< relref "../../administration/configuration/#smtp" >}})
in the Grafana config. Email notifications will upload an image of the alert graph to an
external image destination if available or fallback to attaching the image to the email.
Be aware that if you use the `local` image storage email servers and clients might not be
able to access the image.
> **Note:** Template variables are not supported in email alerts.
| Setting | Description |
| ------------ | -------------------------------------------------------------------------------------------- |
| Single email | Send a single email to all recipients. Disabled per default. |
| Addresses | Email addresses to recipients. You can enter multiple email addresses using a ";" separator. |
### Slack
{{< figure class="float-right" max-width="40%" src="/static/img/docs/v4/slack_notification.png" caption="Alerting Slack Notification" >}}
To set up Slack, you need to configure an incoming Slack webhook URL. You can follow
[Sending messages using Incoming Webhooks](https://api.slack.com/incoming-webhooks) on how to do that. If you want to include screenshots of the
firing alerts in the Slack messages you have to configure either the [external image destination](#external-image-store)
in Grafana or a bot integration via Slack Apps. [Follow Slack's guide to set up a bot integration](https://api.slack.com/bot-users) and use the token
provided, which starts with "xoxb".
| Setting | Description |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Url | Slack incoming webhook URL, or eventually the [chat.postMessage](https://api.slack.com/methods/chat.postMessage) Slack API endpoint. |
| Username | Set the username for the bot's message. |
| Recipient | Allows you to override the Slack recipient. You must either provide a channel Slack ID, a user Slack ID, a username reference (@&lt;user&gt;, all lowercase, no whitespace), or a channel reference (#&lt;channel&gt;, all lowercase, no whitespace). If you use the `chat.postMessage` Slack API endpoint, this is required. |
| Icon emoji | Provide an emoji to use as the icon for the bot's message. Ex :smile: |
| Icon URL | Provide a URL to an image to use as the icon for the bot's message. |
| Mention Users | Optionally mention one or more users in the Slack notification sent by Grafana. You have to refer to users, comma-separated, via their corresponding Slack IDs (which you can find by clicking the overflow button on each user's Slack profile). |
| Mention Groups | Optionally mention one or more groups in the Slack notification sent by Grafana. You have to refer to groups, comma-separated, via their corresponding Slack IDs (which you can get from each group's Slack profile URL). |
| Mention Channel | Optionally mention either all channel members or just active ones. |
| Token | If provided, Grafana will upload the generated image via Slack's file.upload API method, not the external image destination. If you use the `chat.postMessage` Slack API endpoint, this is required. |
If you are using the token for a slack bot, then you have to invite the bot to the channel you want to send notifications and add the channel to the recipient field.
### Opsgenie
To setup Opsgenie you will need an API Key and the Alert API Url. These can be obtained by configuring a new [Grafana Integration](https://docs.opsgenie.com/docs/grafana-integration).
| Setting | Description |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Alert API URL | The API URL for your Opsgenie instance. This will normally be either `https://api.opsgenie.com` or, for EU customers, `https://api.eu.opsgenie.com`. |
| API Key | The API Key as provided by Opsgenie for your configured Grafana integration. |
| Override priority | Configures the alert priority using the `og_priority` tag. The `og_priority` tag must have one of the following values: `P1`, `P2`, `P3`, `P4`, or `P5`. Default is `False`. |
| Send notification tags as | Specify how you would like [Notification Tags]({{< relref "create-alerts.md/#notifications" >}}) delivered to Opsgenie. They can be delivered as `Tags`, `Extra Properties` or both. Default is Tags. See note below for more information. |
> **Note:** When notification tags are sent as `Tags` they are concatenated into a string with a `key:value` format. If you prefer to receive the notifications tags as key/values under Extra Properties in Opsgenie then change the `Send notification tags as` to either `Extra Properties` or `Tags & Extra Properties`.
### PagerDuty
To set up PagerDuty, all you have to do is to provide an integration key.
| Setting | Description |
| ---------------------- | ----------------------------------------------------------------------------------------------- |
| Integration Key | Integration key for PagerDuty. |
| Severity | Level for dynamic notifications, default is `critical` (1) |
| Auto resolve incidents | Resolve incidents in PagerDuty once the alert goes back to ok |
| Message in details | Removes the Alert message from the PD summary field and puts it into custom details instead (2) |
> **Note:** The tags `Severity`, `Class`, `Group`, `dedup_key`, and `Component` have special meaning in the [Pagerduty Common Event Format - PD-CEF](https://support.pagerduty.com/docs/pd-cef). If an alert panel defines these tag keys, then they are transposed to the root of the event sent to Pagerduty. This means they will be available within the Pagerduty UI and Filtering tools. A Severity tag set on an alert overrides the global Severity set on the notification channel if it's a valid level.
> Using Message In Details will change the structure of the `custom_details` field in the PagerDuty Event.
> This might break custom event rules in your PagerDuty rules if you rely on the fields in `payload.custom_details`.
> Move any existing rules using `custom_details.myMetric` to `custom_details.queries.myMetric`.
> This behavior will become the default in a future version of Grafana.
> **Note:** The `dedup_key` tag overrides the Grafana-generated `dedup_key` with a custom key.
> **Note:** The `state` tag overrides the current alert state inside the `custom_details` payload.
> **Note:** Grafana uses the `Events API V2` integration. This can be configured for each service.
### VictorOps
To configure VictorOps, provide the URL from the Grafana Integration and substitute `$routing_key` with a valid key.
> **Note:** The tag `Severity` has special meaning in the [VictorOps Incident Fields](https://help.victorops.com/knowledge-base/incident-fields-glossary/). If an alert panel defines this key, then it replaces the `message_type` in the root of the event sent to VictorOps.
### Pushover
To set up Pushover, you must provide a user key and an API token. Refer to [What is Pushover and how do I use it](https://support.pushover.net/i7-what-is-pushover-and-how-do-i-use-it) for instructions on how to generate them.
| Setting | Description |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| API Token | Application token |
| User key(s) | A comma-separated list of user keys |
| Device(s) | A comma-separated list of devices |
| Priority | The priority alerting nottifications are sent |
| OK priority | The priority OK notifications are sent; if not set, then OK notifications are sent with the priority set for alerting notifications |
| Retry | How often (in seconds) the Pushover servers send the same notification to the user. (minimum 30 seconds) |
| Expire | How many seconds your notification will continue to be retried for (maximum 86400 seconds) |
| Alerting sound | The sound for alerting notifications |
| OK sound | The sound for OK notifications |
### Webhook
The webhook notification is a simple way to send information about a state change over HTTP to a custom endpoint.
Using this notification you could integrate Grafana into a system of your choosing.
Example json body:
```json
{
"dashboardId": 1,
"evalMatches": [
{
"value": 1,
"metric": "Count",
"tags": {}
}
],
"imageUrl": "https://grafana.com/assets/img/blog/mixed_styles.png",
"message": "Notification Message",
"orgId": 1,
"panelId": 2,
"ruleId": 1,
"ruleName": "Panel Title alert",
"ruleUrl": "http://localhost:3000/d/hZ7BuVbWz/test-dashboard?fullscreen\u0026edit\u0026tab=alert\u0026panelId=2\u0026orgId=1",
"state": "alerting",
"tags": {
"tag name": "tag value"
},
"title": "[Alerting] Panel Title alert"
}
```
- **state** - The possible values for alert state are: `ok`, `paused`, `alerting`, `pending`, `no_data`.
### DingDing/DingTalk
DingTalk supports the following "message type": `text`, `link` and `markdown`. Only the `link` message type is supported. Refer to the [configuration instructions](https://developers.dingtalk.com/document/app/custom-robot-access) in Chinese language.
In DingTalk PC Client:
1. Click "more" icon on upper right of the panel.
2. Click "Robot Manage" item in the pop menu, there will be a new panel call "Robot Manage".
3. In the "Robot Manage" panel, select "customized: customized robot with Webhook".
4. In the next new panel named "robot detail", click "Add" button.
5. In "Add Robot" panel, input a nickname for the robot and select a "message group" which the robot will join in. click "next".
6. There will be a Webhook URL in the panel, looks like this: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxx. Copy this URL to the Grafana DingTalk setting page and then click "finish".
### Discord
To set up Discord, you must create a Discord channel webhook. For instructions on how to create the channel, refer to
[Intro to Webhooks](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks).
| Setting | Description |
| ------------------------------ | ----------------------------------------------------------------------------------------------------- |
| Webhook URL | Discord webhook URL. |
| Message Content | Mention a group using @ or a user using <@ID> when notifying in a channel. |
| Avatar URL | Optionally, provide a URL to an image to use as the avatar for the bot's message. |
| Use Discord's Webhook Username | Use the username configured in Discord's webhook settings. Otherwise, the username will be 'Grafana.' |
Alternately, use the [Slack](#slack) notifier by appending `/slack` to a Discord webhook URL.
### Kafka
Notifications can be sent to a Kafka topic from Grafana using the [Kafka REST Proxy](https://docs.confluent.io/1.0/kafka-rest/docs/index.html).
There are a couple of configuration options which need to be set up in Grafana UI under Kafka Settings:
1. Kafka REST Proxy endpoint.
1. Kafka Topic.
Once these two properties are set, you can send the alerts to Kafka for further processing or throttling.
### Google Hangouts Chat
Notifications can be sent by setting up an incoming webhook in Google Hangouts chat. For more information about configuring a webhook, refer to [webhooks](https://developers.google.com/hangouts/chat/how-tos/webhooks).
### Prometheus Alertmanager
Alertmanager handles alerts sent by client applications such as Prometheus server or Grafana. It takes care of deduplicating, grouping, and routing them to the correct receiver. Grafana notifications can be sent to Alertmanager via a simple incoming webhook. Refer to the official [Prometheus Alertmanager documentation](https://prometheus.io/docs/alerting/alertmanager) for configuration information.
> **Caution:** In case of a high-availability setup, do not load balance traffic between Grafana and Alertmanagers to keep coherence between all your Alertmanager instances. Instead, point Grafana to a list of all Alertmanagers, by listing their URLs comma-separated in the notification channel configuration.
### Sensu Go
Grafana alert notifications can be sent to [Sensu](<(https://sensu.io)>) Go as events via the API. This operation requires an API key. For information on creating this key, refer to [Sensu Go documentation](https://docs.sensu.io/sensu-go/latest/operations/control-access/use-apikeys/#api-key-authentication).
## Enable images in notifications {#external-image-store}
Grafana can render the panel associated with the alert rule as a PNG image and include that in the notification. Read more about the requirements and how to configure
[image rendering]({{< relref "../../image-rendering/" >}}).
You must configure an [external image storage provider]({{< relref "../../administration/configuration/#external-image-storage" >}}) in order to receive images in alert notifications. If your notification channel requires that the image be publicly accessible (e.g. Slack, PagerDuty), configure a provider which uploads the image to a remote image store like Amazon S3, Webdav, Google Cloud Storage, or Azure Blob Storage. Otherwise, the local provider can be used to serve the image directly from Grafana.
Notification services which need public image access are marked as 'external only'.
## Configure the link back to Grafana from alert notifications
All alert notifications contain a link back to the triggered alert in the Grafana instance.
This URL is based on the [domain]({{< relref "../../administration/configuration/#domain" >}}) setting in Grafana.
## Notification templating
> **Note:** Alert notification templating is only available in Grafana v7.4 and above.
The alert notification template feature allows you to take the [label]({{< relref "../../basics/timeseries-dimensions.md#labels" >}}) value from an alert query and [inject that into alert notifications]({{< relref "./add-notification-template.md" >}}).
docs/sources/alerting/old-alerting/pause-an-alert-rule.md
-15
View File
@@ -1,15 +0,0 @@
+++
title = "Pause alert rule"
description = "Pause an existing alert rule"
keywords = ["grafana", "alerting", "guide", "rules", "view"]
weight = 400
aliases = ["/docs/grafana/latest/alerting/pause-an-alert-rule/"]
+++
# Pause an alert rule
Pausing the evaluation of an alert rule can sometimes be useful. For example, during a maintenance window, pausing alert rules can avoid triggering a flood of alerts.
1. In the Grafana side bar, hover your cursor over the Alerting (bell) icon and then click **Alert Rules**. All configured alert rules are listed, along with their current state.
1. Find your alert in the list, and click the **Pause** icon on the right. The **Pause** icon turns into a **Play** icon.
1. Click the **Play** icon to resume evaluation of your alert.
docs/sources/alerting/old-alerting/troubleshoot-alerts.md
-42
View File
@@ -1,42 +0,0 @@
+++
title = "Troubleshoot alerts"
description = "Troubleshoot alert rules"
keywords = ["grafana", "alerting", "guide", "rules", "troubleshoot"]
weight = 500
aliases = ["/docs/grafana/latest/alerting/troubleshoot-alerts/"]
+++
# Troubleshoot alerts
If alerts are not behaving as you expect, here are some steps you can take to troubleshoot and figure out what is going wrong.
![Test Rule](/static/img/docs/v4/alert_test_rule.png)
The first level of troubleshooting you can do is click **Test Rule**. You will get result back that you can expand to the point where you can see the raw data that was returned from your query.
Further troubleshooting can also be done by inspecting the grafana-server log. If it's not an error or for some reason the log does not say anything you can enable debug logging for some relevant components. This is done in Grafana's ini config file.
Example showing loggers that could be relevant when troubleshooting alerting.
```ini
[log]
filters = alerting.scheduler:debug \
alerting.engine:debug \
alerting.resultHandler:debug \
alerting.evalHandler:debug \
alerting.evalContext:debug \
alerting.extractor:debug \
alerting.notifier:debug \
alerting.notifier.slack:debug \
alerting.notifier.pagerduty:debug \
alerting.notifier.email:debug \
alerting.notifier.webhook:debug \
tsdb.graphite:debug \
tsdb.prometheus:debug \
tsdb.opentsdb:debug \
tsdb.influxdb:debug \
tsdb.elasticsearch:debug \
tsdb.elasticsearch.client:debug \
```
If you want to log raw query sent to your TSDB and raw response in log you also have to set grafana.ini option `app_mode` to `development`.
docs/sources/alerting/old-alerting/view-alerts.md
-20
View File
@@ -1,20 +0,0 @@
+++
title = "View alerts"
description = "View existing alert rules"
keywords = ["grafana", "alerting", "guide", "rules", "view"]
weight = 400
aliases = ["/docs/grafana/latest/alerting/view-alerts/"]
+++
# View existing alert rules
Grafana stores individual alert rules in the panels where they are defined, but you can also view a list of all existing alert rules and their current state.
In the Grafana side bar, hover your cursor over the Alerting (bell) icon and then click **Alert Rules**. All configured alert rules are listed, along with their current state.
You can do several things while viewing alerts.
- **Filter alerts by name -** Type an alert name in the **Search alerts** field.
- **Filter alerts by state -** In **States**, select which alert states you want to see. All others will be hidden.
- **Pause or resume an alert -** Click the **Pause** or **Play** icon next to the alert to pause or resume evaluation. See [Pause an alert rule]({{< relref "pause-an-alert-rule.md" >}}) for more information.
- **Access alert rule settings -** Click the alert name or the **Edit alert rule** (gear) icon. Grafana opens the Alert tab of the panel where the alert rule is defined. This is helpful when an alert is firing but you don't know which panel it is defined in.
docs/sources/alerting/opt-in.md
+41
View File
@@ -0,0 +1,41 @@
+++
title = "Opt-in to Grafana alerting"
description = "Enable Grafana alerts"
weight = 113
aliases = ["/docs/grafana/latest/alerting/unified-alerting/opt-in/"]
+++
# Opt-in to Grafana alerting in OSS
Grafana alerting is enabled by default for new Cloud and OSS installations.
- For existing OSS installations that use legacy dashboard alerts, unified alerting is still an opt-in feature.
- For existing Grafana Cloud users, contact customer support to enable Grafana alerting for your Cloud stack.
## Before you begin
We recommend that you backup Grafana's database. If you are using PostgreSQL as the backend database, then the minimum required version is 9.5.
## Enable Grafana alerting
To enable Grafana alerts:
1. In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the [unified alerts]({{< relref "../administration/configuration.md#unified_alerting" >}}) section.
2. Set the `enabled` property to `true`.
3. Next, for [legacy dashboard alerting]({{< relref "../administration/configuration.md#alerting" >}}), set the `enabled` flag to `false`.
4. Restart Grafana for the configuration changes to take effect.
> **Note:** The `ngalert` toggle previously used to enable or disable Grafana alerting is no longer available.
Before v8.2, notification logs and silences were stored on a disk. If you did not use persistent disks, you would have lost any configured silences and logs on a restart, resulting in unwanted or duplicate notifications. We no longer require the use of a persistent disk. Instead, the notification logs and silences are stored regularly (every 15 minutes). If you used the file-based approach, Grafana reads the existing file and persists it eventually.
## Disable Grafana alerts
To disable Grafana alerts and roll back to legacy dashboard alerting:
1. In your custom configuration file ($WORKING_DIR/conf/custom.ini), go to the [Grafana alerting]({{< relref "../administration/configuration.md#unified_alerting" >}}) section.
1. Set the `enabled` property to `false`.
1. For [legacy dashboard alerting]({{< relref "../administration/configuration.md#alerting" >}}), set the `enabled` flag to `true`.
1. Restart Grafana for the configuration changes to take effect.
> **Note:** Switching from one flavor of alerting to another can result in data loss. This is applicable to the fresh installation as well as upgraded setups.
docs/sources/alerting/unified-alerting/silences.md → docs/sources/alerting/silences.md
+3 -2
View File
@@ -3,6 +3,7 @@ title = "Silences"
description = "Silences alert notifications"
keywords = ["grafana", "alerting", "silence", "mute"]
weight = 400
aliases = ["/docs/grafana/latest/alerting/unified-alerting/silences/"]
+++
# Silences
@@ -11,7 +12,7 @@ Use silences to stop notifications from one or more alerting rules. Silences do
Silences do not prevent alert rules from being evaluated. They also do not stop alert instances being shown in the user interface. Silences only prevent notifications from being created.
You can configure Grafana managed silences as well as silences for an [external Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}). For more information, see [Alertmanager]({{< relref "./fundamentals/alertmanager.md" >}}).
You can configure Grafana managed silences as well as silences for an [external Alertmanager data source]({{< relref "../datasources/alertmanager.md" >}}). For more information, see [Alertmanager]({{< relref "./fundamentals/alertmanager.md" >}}).
> **Note:** Before Grafana v8.2, the configuration of the embedded Alertmanager was shared across organisations. Users of Grafana 8.0 and 8.1 are advised to use the new Grafana 8 Alerts only if they have one organisation. Otherwise, silences for the Grafana managed alerts will be visible by all organizations.
@@ -61,4 +62,4 @@ When linking to a silence form, provide the default matching labels and comment
For example, to link to silence form with matching labels `severity=critical` & `cluster!~europe-.*` and comment `Silence critical EU alerts`, create a URL `https://mygrafana/alerting/silence/new?matchers=severity%3Dcritical%2Ccluster!~europe-*&comment=Silence%20critical%20EU%20alert`.
To link to a new silence page for an [external Alertmanager]({{< relref "../../datasources/alertmanager.md" >}}), add a `alertmanager` query parameter with the Alertmanager data source name.
To link to a new silence page for an [external Alertmanager]({{< relref "../datasources/alertmanager.md" >}}), add a `alertmanager` query parameter with the Alertmanager data source name.
docs/sources/alerting/unified-alerting/_index.md
-28
View File
@@ -1,28 +0,0 @@
+++
title = "Grafana alerts"
aliases = ["/docs/grafana/latest/alerting/metrics/"]
weight = 113
+++
# Overview of Grafana alerting
Grafana 8.0 has new and improved alerting that centralizes alerting information in a single, searchable view. It is enabled by default for all new OSS instances, and is an [opt-in]({{< relref "./opt-in.md" >}}) feature for older installations that still use legacy dashboard alerting. We encourage you to create issues in the Grafana GitHub repository for bugs found while testing Grafana alerting. See also, [What's New with Grafana alerting]({{< relref "./difference-old-new.md" >}}).
> Refer to [Fine-grained access control]({{< relref "../../enterprise/access-control/_index.md" >}}) in Grafana Enterprise to learn more about controlling access to alerts using fine-grained permissions.
When Grafana alerting is enabled, you can:
- [Create Grafana managed alerting rules]({{< relref "alerting-rules/create-grafana-managed-rule.md" >}})
- [Create Grafana Mimir or Loki managed alerting rules]({{< relref "alerting-rules/create-mimir-loki-managed-rule.md" >}})
- [View existing alerting rules and manage their current state]({{< relref "alerting-rules/rule-list.md" >}})
- [View the state and health of alerting rules]({{< relref "./fundamentals/state-and-health.md" >}})
- [Add or edit an alert contact point]({{< relref "./contact-points.md" >}})
- [Add or edit notification policies]({{< relref "./notifications/_index.md" >}})
- [Add or edit silences]({{< relref "./silences.md" >}})
Before you begin using Grafana alerting, we recommend that you familiarize yourself with some [basic concepts]({{< relref "./fundamentals/_index.md" >}}) of Grafana alerting.
## Limitations
- The Grafana alerting system can retrieve rules from all available Prometheus, Loki, and Alertmanager data sources. It might not be able to fetch rules from other supported data sources.
- We aim to support the latest two minor versions of both Prometheus and Alertmanager. We cannot guarantee that older versions will work. As an example, if the current Prometheus version is `2.31.1`, we support >= `2.29.0`.