diff --git a/docs/sources/alerting/alerting-rules/create-grafana-managed-rule.md b/docs/sources/alerting/alerting-rules/create-grafana-managed-rule.md index a26c9c2ca76..f7b5a38d753 100644 --- a/docs/sources/alerting/alerting-rules/create-grafana-managed-rule.md +++ b/docs/sources/alerting/alerting-rules/create-grafana-managed-rule.md @@ -171,16 +171,16 @@ Complete the following steps to set up labels and notifications. 2. You can also optionally select a mute timing as well as groupings and timings to define when not to send notifications. - {{% admonition type="note" %}} - An auto-generated notification policy is generated. Only admins can view these auto-generated policies from the **Notification policies** list view. Any changes have to be made in the alert rules form. {{% /admonition %}} + {{< admonition type="note" >}} + An auto-generated notification policy is generated. Only admins can view these auto-generated policies from the **Notification policies** list view. Any changes have to be made in the alert rules form. {{< /admonition >}} **Use notification policy** 3. Choose this option to use the notification policy tree to direct your notifications. - {{% admonition type="note" %}} + {{< admonition type="note" >}} All alert rules and instances, irrespective of their labels, match the default notification policy. If there are no nested policies, or no nested policies match the labels in the alert rule or alert instance, then the default notification policy is the matching policy. - {{% /admonition %}} + {{< /admonition >}} 4. Preview your alert instance routing set up. diff --git a/docs/sources/alerting/configure-notifications/create-notification-policy.md b/docs/sources/alerting/configure-notifications/create-notification-policy.md index 55d0e331926..1cdd8ddccff 100644 --- a/docs/sources/alerting/configure-notifications/create-notification-policy.md +++ b/docs/sources/alerting/configure-notifications/create-notification-policy.md @@ -18,7 +18,7 @@ labels: - enterprise - oss title: Configure notification policies -weight: 430 +weight: 420 --- # Configure notification policies diff --git a/docs/sources/alerting/configure-notifications/create-silence.md b/docs/sources/alerting/configure-notifications/create-silence.md index 874cd40df51..ab6cfd92877 100644 --- a/docs/sources/alerting/configure-notifications/create-silence.md +++ b/docs/sources/alerting/configure-notifications/create-silence.md @@ -19,11 +19,11 @@ labels: - cloud - enterprise - oss -title: Manage silences +title: Configure silences weight: 440 --- -# Manage silences +# Configure silences Silences stop notifications from getting created and last for only a specified window of time. diff --git a/docs/sources/alerting/configure-notifications/mute-timings.md b/docs/sources/alerting/configure-notifications/mute-timings.md index 2c4d9a52dc5..c2c680b1c3c 100644 --- a/docs/sources/alerting/configure-notifications/mute-timings.md +++ b/docs/sources/alerting/configure-notifications/mute-timings.md @@ -17,11 +17,11 @@ labels: - cloud - enterprise - oss -title: Create mute timings +title: Configure mute timings weight: 450 --- -# Create mute timings +# Configure mute timings A mute timing is a recurring interval of time when no new notifications for a policy are generated or sent. Use them to prevent alerts from firing a specific and reoccurring period, for example, a regular maintenance period. diff --git a/docs/sources/alerting/configure-notifications/template-notifications/_index.md b/docs/sources/alerting/configure-notifications/template-notifications/_index.md index c096b044ca1..d160522b345 100644 --- a/docs/sources/alerting/configure-notifications/template-notifications/_index.md +++ b/docs/sources/alerting/configure-notifications/template-notifications/_index.md @@ -13,13 +13,13 @@ labels: - cloud - enterprise - oss -title: Customize notifications -weight: 420 +title: Configure notification messages +weight: 430 --- -# Customize notifications +# Configure notification messages -Customize your notifications with notifications templates. +Customize the content of your notifications with notifications templates. You can use notification templates to change the title, message, and format of the message in your notifications. diff --git a/docs/sources/alerting/fundamentals/_index.md b/docs/sources/alerting/fundamentals/_index.md index 45c1427cf80..3066c268776 100644 --- a/docs/sources/alerting/fundamentals/_index.md +++ b/docs/sources/alerting/fundamentals/_index.md @@ -18,33 +18,42 @@ weight: 100 Whether you’re just starting out or you're a more experienced user of Grafana Alerting, learn more about the fundamentals and available features that help you create, manage, and respond to alerts; and improve your team’s ability to resolve issues quickly. -## Principles - -In Prometheus-based alerting systems, you have an alert generator that creates alerts and an alert receiver that receives alerts. For example, Prometheus is an alert generator and is responsible for evaluating alert rules, while Alertmanager is an alert receiver and is responsible for grouping, inhibiting, silencing, and sending notifications about firing and resolved alerts. - -Grafana Alerting is built on the Prometheus model of designing alerting systems. It has an internal alert generator responsible for scheduling and evaluating alert rules, as well as an internal alert receiver responsible for grouping, inhibiting, silencing, and sending notifications. Grafana doesn’t use Prometheus as its alert generator because Grafana Alerting needs to work with many other data sources in addition to Prometheus. However, it does use Alertmanager as its alert receiver. - -Alerts are sent to the alert receiver where they are routed, grouped, inhibited, silenced and notified. In Grafana Alerting, the default alert receiver is the Alertmanager embedded inside Grafana, and is referred to as the Grafana Alertmanager. However, you can use other Alertmanagers too, and these are referred to as [External Alertmanagers][external-alertmanagers]. - The following diagram gives you an overview of Grafana Alerting and introduces you to some of the fundamental features that are the principles of how Grafana Alerting works. {{< figure src="/media/docs/alerting/how-alerting-works.png" max-width="750px" caption="How Alerting works" >}} +## How it works at a glance + +- Grafana alerting periodically queries data sources and evaluates the condition defined in the alert rule +- If the condition is breached, an alert instance fires +- Firing instances are routed to notification policies based on matching labels +- Notifications are sent out to the contact points specified in the notification policy + ## Fundamentals +The following concepts are key to your understanding of how Grafana Alerting works. + ### Alert rules -An alert rule is a set of criteria that determine when an alert should fire. It consists of one or more queries and expressions, a condition which needs to be met, an interval which determines how often the alert rule is evaluated, and a duration over which the condition must be met for an alert to fire. +An alert rule consists of one or more queries and expressions that select the data you want to measure. It also contains a condition, which is the threshold that an alert rule must meet or exceed in order to fire. -Alert rules are evaluated over their interval, and each alert rule can have zero, one, or any number of alerts firing at a time. The state of the alert rule is determined by its most "severe" alert, which can be one of Normal, Pending, or Firing. For example, if at least one of an alert rule's alerts are firing then the alert rule is also firing. The health of an alert rule is determined by the status of its most recent evaluation. These can be OK, Error, and NoData. +Add annotations to your alert rule to provide additional information about the alert rule and add labels to uniquely identify your alert rule and configure alert routing. Labels link alert rules to notification policies, so you can easily manage which policy should handle which alerts and who gets notified. -A very important feature of alert rules is that they support custom annotations and labels. These allow you to instrument alerts with additional metadata such as summaries and descriptions, and add additional labels to route alerts to specific notification policies. +Once alert rules are created, they go through various states and transitions. An alert rule can produce multiple alert instances - one alert instance for each time series. -### Alerts +The alert rule state is determined by the “worst case” state of the alert instances produced and the states can be Normal, Pending, or Firing. For example, if one alert instance is firing, the alert rule state will also be firing. -Alerts are uniquely identified by sets of key/value pairs called Labels. Each key is a label name and each value is a label value. For example, one alert might have the labels `foo=bar` and another alert might have the labels `foo=baz`. An alert can have many labels such as `foo=bar,bar=baz` but it cannot have the same label twice such as `foo=bar,foo=baz`. Two alerts cannot have the same labels either, and if two alerts have the same labels such as `foo=bar,bar=baz` and `foo=bar,bar=baz` then one of the alerts will be discarded. Alerts are resolved when the condition in the alert rule is no longer met, or the alert rule is deleted. +The alert rule health is determined by the status of the evaluation of the alert rule, which can be Ok, Error, and NoData. -In Grafana Managed Alerts, alerts can be in Normal, Pending, Alerting, No Data or Error states. In Datasource Managed Alerts, such as Mimir and Loki, alerts can be in Normal, Pending and Alerting, but not NoData or Error. +### Labels and states + +Alert rules are uniquely identified by sets of key/value pairs called labels. Each key is a label name and each value is a label value. For example, one alert might have the labels `foo=bar` and another alert rule might have the labels `foo=baz`. An alert rule can have many labels such as `foo=bar,bar=baz`, but it cannot have the same label twice such as `foo=bar,foo=baz`. Two alert rules cannot have the same labels either, and if two alert rules have the same labels such as `foo=bar,bar=baz` and `foo=bar,bar=baz` then one of the alerts will be discarded. Firing alerts are resolved when the condition in the alert rule is no longer met, or the alert rule is deleted. + +In Grafana-managed alert rules, alert rules can be in Normal, Pending, Alerting, No Data or Error states. In datasource-managed alert rules, such as Mimir and Loki, alert rules can be in Normal, Pending and Alerting, but not NoData or Error. + +### Alert instances + +For Grafana-managed alert rules, multiple alert instances can be created as a result of one alert rule (also known as a multi-dimensional alerting) and they can be in Normal, Pending, Alerting, No Data, Error states. For Mimir or Loki-managed alert rules, alert instances are only created when the threshold condition defined in an alert rule is breached. ### Contact points @@ -68,6 +77,14 @@ Silences and mute timings allow you to pause notifications for specific alerts o You can create your alerting resources (alert rules, notification policies, and so on) in the Grafana UI; configmaps, files and configuration management systems using file-based provisioning; and in Terraform using API-based provisioning. +## Principles + +In Prometheus-based alerting systems, you have an alert generator that creates alerts and an alert receiver that receives alerts. For example, Prometheus is an alert generator and is responsible for evaluating alert rules, while Alertmanager is an alert receiver and is responsible for grouping, inhibiting, silencing, and sending notifications about firing and resolved alerts. + +Grafana Alerting is built on the Prometheus model of designing alerting systems. It has an internal alert generator responsible for scheduling and evaluating alert rules, as well as an internal alert receiver responsible for grouping, inhibiting, silencing, and sending notifications. Grafana doesn’t use Prometheus as its alert generator because Grafana Alerting needs to work with many other data sources in addition to Prometheus. However, it does use Alertmanager as its alert receiver. + +Alerts are sent to the alert receiver where they are routed, grouped, inhibited, silenced and notified. In Grafana Alerting, the default alert receiver is the Alertmanager embedded inside Grafana, and is referred to as the Grafana Alertmanager. However, you can use other Alertmanagers too, and these are referred to as [External Alertmanagers][external-alertmanagers]. + {{% docs/reference %}} [external-alertmanagers]: "/docs/grafana/ -> /docs/grafana//alerting/set-up/configure-alertmanager" [external-alertmanagers]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/set-up/configure-alertmanager" diff --git a/docs/sources/alerting/fundamentals/alert-rules/_index.md b/docs/sources/alerting/fundamentals/alert-rules/_index.md index a7ed8f0f7a0..40bf58e29f5 100644 --- a/docs/sources/alerting/fundamentals/alert-rules/_index.md +++ b/docs/sources/alerting/fundamentals/alert-rules/_index.md @@ -11,7 +11,7 @@ labels: - enterprise - oss title: Alert rules -weight: 130 +weight: 100 --- # Alert rules diff --git a/docs/sources/alerting/fundamentals/alertmanager.md b/docs/sources/alerting/fundamentals/alertmanager.md index 9bdd9fbb082..3e42d857d3c 100644 --- a/docs/sources/alerting/fundamentals/alertmanager.md +++ b/docs/sources/alerting/fundamentals/alertmanager.md @@ -12,7 +12,7 @@ labels: - enterprise - oss title: Alertmanager -weight: 140 +weight: 150 --- # Alertmanager diff --git a/docs/sources/alerting/fundamentals/annotation-label/_index.md b/docs/sources/alerting/fundamentals/annotation-label/_index.md index 690902b7cc9..65ef622c1f1 100644 --- a/docs/sources/alerting/fundamentals/annotation-label/_index.md +++ b/docs/sources/alerting/fundamentals/annotation-label/_index.md @@ -16,7 +16,7 @@ labels: - enterprise - oss title: Labels and annotations -weight: 120 +weight: 130 --- # Labels and annotations diff --git a/docs/sources/alerting/fundamentals/contact-points/index.md b/docs/sources/alerting/fundamentals/contact-points/index.md index 5494a9b6c0a..4e42886a804 100644 --- a/docs/sources/alerting/fundamentals/contact-points/index.md +++ b/docs/sources/alerting/fundamentals/contact-points/index.md @@ -18,7 +18,7 @@ labels: - enterprise - oss title: Contact points -weight: 150 +weight: 120 --- # Contact points diff --git a/docs/sources/alerting/fundamentals/data-source-alerting.md b/docs/sources/alerting/fundamentals/data-source-alerting.md index cae5474a889..fcec0f23527 100644 --- a/docs/sources/alerting/fundamentals/data-source-alerting.md +++ b/docs/sources/alerting/fundamentals/data-source-alerting.md @@ -7,7 +7,7 @@ labels: - enterprise - oss title: Data sources and Grafana Alerting -weight: 100 +weight: 140 --- # Data sources and Grafana Alerting diff --git a/docs/sources/alerting/fundamentals/evaluate-grafana-alerts.md b/docs/sources/alerting/fundamentals/evaluate-grafana-alerts.md index c7ec0565b20..a238358029d 100644 --- a/docs/sources/alerting/fundamentals/evaluate-grafana-alerts.md +++ b/docs/sources/alerting/fundamentals/evaluate-grafana-alerts.md @@ -10,7 +10,7 @@ labels: - enterprise - oss title: Alerting on numeric data -weight: 110 +weight: 160 --- # Alerting on numeric data diff --git a/docs/sources/alerting/fundamentals/notification-policies/_index.md b/docs/sources/alerting/fundamentals/notification-policies/_index.md index 56dc8e06bd3..ef689ec6fd5 100644 --- a/docs/sources/alerting/fundamentals/notification-policies/_index.md +++ b/docs/sources/alerting/fundamentals/notification-policies/_index.md @@ -11,7 +11,7 @@ labels: - enterprise - oss title: Notifications -weight: 160 +weight: 110 --- # Notifications