Docs: Sync all doc changes in main to v8.0.x (#35333)
This commit is contained in:
Torkel Ödegaard
@@ -19,10 +19,10 @@ Currently only the graph panel visualization supports alerts.
|
||||
|
||||
You can perform the following tasks for alerts:
|
||||
|
||||
- [Add or edit an alert notification channel]({{< relref "notifications.md" >}})
|
||||
- [Create an alert rule]({{< relref "create-alerts.md" >}})
|
||||
- [View existing alert rules and their current state]({{< relref "view-alerts.md" >}})
|
||||
- [Test alert rules and troubleshoot]({{< relref "troubleshoot-alerts.md" >}})
|
||||
- [Add or edit an alert notification channel]({{< relref "./old-alerting/notifications.md" >}})
|
||||
- [Create an alert rule]({{< relref "./old-alerting/create-alerts.md" >}})
|
||||
- [View existing alert rules and their current state]({{< relref "./old-alerting/view-alerts.md" >}})
|
||||
- [Test alert rules and troubleshoot]({{< relref "./old-alerting/troubleshoot-alerts.md" >}})
|
||||
|
||||
## Clustering
|
||||
|
||||
@@ -39,7 +39,7 @@ The actual notifications are configured and shared between multiple alerts.
|
||||
Alert rules are evaluated in the Grafana backend in a scheduler and query execution engine that is part
|
||||
of core Grafana. Alert rules can query only backend data sources with alerting enabled. Such data sources are:
|
||||
- builtin or developed and maintained by grafana, such as: `Graphite`, `Prometheus`, `Loki`, `InfluxDB`, `Elasticsearch`,
|
||||
`Google Cloud Monitoring`, `Cloudwatch`, `Azure Monitor`, `MySQL`, `PostgreSQL`, `MSSQL`, `OpenTSDB`, `Oracle`, and `Azure Data Explorer`
|
||||
`Google Cloud Monitoring`, `Cloudwatch`, `Azure Monitor`, `MySQL`, `PostgreSQL`, `MSSQL`, `OpenTSDB`, `Oracle`, and `Azure Data Explorer`
|
||||
- any community backend data sources with alerting enabled (`backend` and `alerting` properties are set in the [plugin.json]({{< relref "../developers/plugins/metadata.md" >}}))
|
||||
|
||||
## Metrics from the alert engine
|
||||
|
||||
@@ -1,18 +1,21 @@
|
||||
+++
|
||||
title = "Difference between new and old alerts"
|
||||
description = "Pause an existing alert rule"
|
||||
keywords = ["grafana", "alerting", "guide", "rules", "view"]
|
||||
weight = 400
|
||||
title = "What's New with Grafana 8 Alerts"
|
||||
description = "What's New with Grafana 8 Alerts"
|
||||
keywords = ["grafana", "alerting", "guide"]
|
||||
weight = 112
|
||||
+++
|
||||
|
||||
# Difference between new and old alerts
|
||||
# Differences between Grafana 8 alerts and legacy alerts
|
||||
The Alerts released with Grafana 8.0 are an opt-in feature that centralizes alerting information for Grafana managed alerts and alerts from Prometheus-compatible datasources in one UI and API. You are able to create and edit alerting rules for Grafana managed alerts, Cortex alerts, and Loki alerts as well as see alerting information from prometheus-compatible datasources in a single, searchable view.
|
||||
|
||||
The `ngalert` feature toggle enables the beta version of our new alerting system.
|
||||
## Multi-dimensional alerting
|
||||
Create alerts that will give you system-wide visibility with a single alerting rule. With Grafana 8 alerts, you are able to generate multiple alert instances from a single rule eg. creating a rule to monitor disk usage for multiple mount points on a single host. The evaluation engine is able to return multiple time series from a single query. Each time series is identified by its label set.
|
||||
|
||||
>**Note:** It is recommended to backup Grafana's database before enabling this feature.
|
||||
## Create alerts outside of Dashboards
|
||||
Grafana legacy alerts were tied to a dashboard. Grafana 8 Alerts allow you to create queries and expressions that can combine data from multiple sources, in unique ways. You are still able to link dashboards and panels to alerting rules, allowing you to quickly troubleshoot the system under observation, by linking a dashboard and/or panel ID to the alerting rule.
|
||||
|
||||
When the feature flag is enabled, dashboard alerting is disabled and dashboard alerts are migrated into the system. Going to "Alert List" will take you to the new system.
|
||||
## Create Loki and Cortex alerting rules
|
||||
With Grafana 8 Alerts you are able to manage your Loki and Cortex alerting rules using the same UI and API as your Grafana managed alerts.
|
||||
|
||||
Once you disable the new alters, all migrated and newly created alerts in the new system are deleted, and dashboard alerting will be enabled again.
|
||||
|
||||
During beta, the migration of existing dashboard rules may change.
|
||||
## View and search for alerts from Prometheus
|
||||
You can now display all of your alerting information in one, searchable UI. Alerts for Prometheus compatible datasources are listed below Grafana managed alerts. Search for labels across multiple datasources to quickly find all of the relevant alerts.
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
+++
|
||||
title = "Old Grafana Alerts"
|
||||
aliases = ["/docs/grafana/latest/alerting/rules/", "/docs/grafana/latest/alerting/metrics/"]
|
||||
weight = 114
|
||||
+++
|
||||
|
||||
# Old Grafana alerts
|
||||
|
||||
Alerts allow you to know about problems in your systems moments after they occur. Robust and actionable alerts help you identify and resolve issues quickly, minimizing disruption to your services.
|
||||
|
||||
Alerts have four main components:
|
||||
|
||||
- Alert rule - One or more conditions, the frequency of evaluation, and the (optional) duration that a condition must be met before notifying.
|
||||
- Contact point - A channel for sending notifications when the conditions of an alert rule are met.
|
||||
- Notification policy - A set of matching and grouping criteria used to determine where, and how frequently, to send notifications.
|
||||
- Silences - Date and matching criteria used to silence notifications.
|
||||
|
||||
## Alert tasks
|
||||
|
||||
You can perform the following tasks for alerts:
|
||||
|
||||
- [Create an alert rule]({{< relref "create-alerts.md" >}})
|
||||
- [View existing alert rules and their current state]({{< relref "view-alerts.md" >}})
|
||||
- [Test alert rules and troubleshoot]({{< relref "troubleshoot-alerts.md" >}})
|
||||
- [Add or edit an alert contact point]({{< relref "notifications.md" >}})
|
||||
|
||||
## Clustering
|
||||
|
||||
Currently alerting supports a limited form of high availability. Since v4.2.0 of Grafana, alert notifications are deduped when running multiple servers. This means all alerts are executed on every server but no duplicate alert notifications are sent due to the deduping logic. Proper load balancing of alerts will be introduced in the future.
|
||||
|
||||
## Alert evaluation
|
||||
|
||||
Grafana managed alerts are evaluated by the Grafana backend. Rule evaluations are scheduled, according to the alert rule configuration, and queries are evaluated by an engine that is part of core Grafana.
|
||||
|
||||
Alert rules can only query backend data sources with alerting enabled:
|
||||
- builtin or developed and maintained by grafana: `Graphite`, `Prometheus`, `Loki`, `InfluxDB`, `Elasticsearch`,
|
||||
`Google Cloud Monitoring`, `Cloudwatch`, `Azure Monitor`, `MySQL`, `PostgreSQL`, `MSSQL`, `OpenTSDB`, `Oracle`, and `Azure Data Explorer`
|
||||
- any community backend data sources with alerting enabled (`backend` and `alerting` properties are set in the [plugin.json]({{< relref "../../developers/plugins/metadata.md" >}}))
|
||||
|
||||
## Metrics from the alert engine
|
||||
|
||||
The alert engine publishes some internal metrics about itself. You can read more about how Grafana publishes [internal metrics]({{< relref "../../administration/view-server/internal-metrics.md" >}}).
|
||||
|
||||
Metric Name | Type | Description
|
||||
---------- | ----------- | ----------
|
||||
`alerting.alerts` | gauge | How many alerts by state
|
||||
`alerting.request_duration_seconds` | histogram | Histogram of requests to the Alerting API
|
||||
`alerting.active_configurations` | gauge | The number of active, non default alertmanager configurations for grafana managed alerts
|
||||
`alerting.rule_evaluations_total` | counter | The total number of rule evaluations
|
||||
`alerting.rule_evaluation_failures_total` | counter | The total number of rule evaluation failures
|
||||
`alerting.rule_evaluation_duration_seconds` | summary | The duration for a rule to execute
|
||||
`alerting.rule_group_rules` | gauge | The number of rules
|
||||
@@ -0,0 +1,23 @@
|
||||
+++
|
||||
title = "Alert notification templating"
|
||||
keywords = ["grafana", "documentation", "alerting", "alerts", "notification", "templating"]
|
||||
weight = 110
|
||||
+++
|
||||
|
||||
# Alert notification templating
|
||||
|
||||
You can provide detailed information to alert notification recipients by injecting alert query data into an alert notification. This topic explains how you can use alert query labels in alert notifications.
|
||||
|
||||
Labels that exist from the evaluation of the alert query can be used in the alert rule name and in the alert notification message fields. The alert label data is injected into the notification fields when the alert is in the alerting state. When there are multiple unique values for the same label, the values are comma-separated.
|
||||
|
||||
This topic explains how you can use alert query labels in alert notifications.
|
||||
|
||||
## Adding alert label data into your alert notification
|
||||
|
||||
1. Navigate to the panel you want to add or edit an alert rule for.
|
||||
1. Click on the panel title, and then click **Edit**.
|
||||
1. On the Alert tab, click **Create Alert**. If an alert already exists for this panel, then you can edit the alert directly.
|
||||
1. Refer to the alert query labels in the alert rule name and/or alert notification message field by using the `${Label}` syntax.
|
||||
1. Click **Save** in the upper right corner to save the alert rule and the dashboard.
|
||||
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
+++
|
||||
title = "Create alerts"
|
||||
description = "Configure alert rules"
|
||||
keywords = ["grafana", "alerting", "guide", "rules"]
|
||||
weight = 200
|
||||
+++
|
||||
|
||||
# Create alerts
|
||||
|
||||
Grafana alerting allows you to attach rules to your dashboard panels. When you save the dashboard, Grafana extracts the alert rules into a separate alert rule storage and schedules them for evaluation.
|
||||
|
||||
|
||||
|
||||
In the Alert tab of the graph panel you can configure how often the alert rule should be evaluated and the conditions that need to be met for the alert to change state and trigger its [notifications]({{< relref "notifications.md" >}}).
|
||||
|
||||
Currently only the graph panel supports alert rules.
|
||||
|
||||
## Add or edit an alert rule
|
||||
|
||||
1. Navigate to the panel you want to add or edit an alert rule for, click the title, and then click **Edit**.
|
||||
1. On the Alert tab, click **Create Alert**. If an alert already exists for this panel, then you can just edit the fields on the Alert tab.
|
||||
1. Fill out the fields. Descriptions are listed below in [Alert rule fields](#alert-rule-fields).
|
||||
1. When you have finished writing your rule, click **Save** in the upper right corner to save alert rule and the dashboard.
|
||||
1. (Optional but recommended) Click **Test rule** to make sure the rule returns the results you expect.
|
||||
|
||||
## Delete an alert
|
||||
|
||||
To delete an alert, scroll to the bottom of the alert and then click **Delete**.
|
||||
|
||||
## Alert rule fields
|
||||
|
||||
This section describes the fields you fill out to create an alert.
|
||||
|
||||
### Rule
|
||||
|
||||
- **Name -** Enter a descriptive name. The name will be displayed in the Alert Rules list. This field supports [templating]({{< relref "./add-notification-template.md" >}}).
|
||||
- **Evaluate every -** Specify how often the scheduler should evaluate the alert rule. This is referred to as the _evaluation interval_.
|
||||
- **For -** Specify how long the query needs to violate the configured thresholds before the alert notification triggers.
|
||||
|
||||
You can set a minimum evaluation interval in the `alerting.min_interval_seconds` configuration field, to set a minimum time between evaluations. Refer to [Configuration]({{< relref "../../administration/configuration.md" >}}#min-interval-seconds) for more information.
|
||||
|
||||
> **Caution:** Do not use `For` with the `If no data or all values are null` setting set to `No Data`. The triggering of `No Data` will trigger instantly and not take `For` into consideration. This may also result in that an OK notification not being sent if alert transitions from `No Data -> Pending -> OK`.
|
||||
|
||||
If an alert rule has a configured `For` and the query violates the configured threshold, then it will first go from `OK` to `Pending`. Going from `OK` to `Pending` Grafana will not send any notifications. Once the alert rule has been firing for more than `For` duration, it will change to `Alerting` and send alert notifications.
|
||||
|
||||
Typically, it's always a good idea to use this setting since it's often worse to get false positive than wait a few minutes before the alert notification triggers. Looking at the `Alert list` or `Alert list panels` you will be able to see alerts in pending state.
|
||||
|
||||
Below you can see an example timeline of an alert using the `For` setting. At ~16:04 the alert state changes to `Pending` and after 4 minutes it changes to `Alerting` which is when alert notifications are sent. Once the series falls back to normal the alert rule goes back to `OK`.
|
||||
{{< figure class="float-right" src="/static/img/docs/v54/alerting-for-dark-theme.png" caption="Alerting For" >}}
|
||||
|
||||
{{< figure class="float-right" max-width="40%" src="/static/img/docs/v4/alerting_conditions.png" caption="Alerting Conditions" >}}
|
||||
|
||||
### Conditions
|
||||
|
||||
Currently the only condition type that exists is a `Query` condition that allows you to
|
||||
specify a query letter, time range and an aggregation function.
|
||||
|
||||
#### Query condition example
|
||||
|
||||
```sql
|
||||
avg() OF query(A, 15m, now) IS BELOW 14
|
||||
```
|
||||
|
||||
- `avg()` Controls how the values for **each** series should be reduced to a value that can be compared against the threshold. Click on the function to change it to another aggregation function.
|
||||
- `query(A, 15m, now)` The letter defines what query to execute from the **Metrics** tab. The second two parameters define the time range, `15m, now` means 15 minutes ago to now. You can also do `10m, now-2m` to define a time range that will be 10 minutes ago to 2 minutes ago. This is useful if you want to ignore the last 2 minutes of data.
|
||||
- `IS BELOW 14` Defines the type of threshold and the threshold value. You can click on `IS BELOW` to change the type of threshold.
|
||||
|
||||
The query used in an alert rule cannot contain any template variables. Currently we only support `AND` and `OR` operators between conditions and they are executed serially.
|
||||
For example, we have 3 conditions in the following order:
|
||||
_condition:A(evaluates to: TRUE) OR condition:B(evaluates to: FALSE) AND condition:C(evaluates to: TRUE)_
|
||||
so the result will be calculated as ((TRUE OR FALSE) AND TRUE) = TRUE.
|
||||
|
||||
We plan to add other condition types in the future, like `Other Alert`, where you can include the state of another alert in your conditions, and `Time Of Day`.
|
||||
|
||||
#### Multiple Series
|
||||
|
||||
If a query returns multiple series, then the aggregation function and threshold check will be evaluated for each series. What Grafana does not do currently is track alert rule state **per series**. This has implications that are detailed in the scenario below.
|
||||
|
||||
- Alert condition with query that returns 2 series: **server1** and **server2**
|
||||
- **server1** series causes the alert rule to fire and switch to state `Alerting`
|
||||
- Notifications are sent out with message: _load peaking (server1)_
|
||||
- In a subsequent evaluation of the same alert rule, the **server2** series also causes the alert rule to fire
|
||||
- No new notifications are sent as the alert rule is already in state `Alerting`.
|
||||
|
||||
So, as you can see from the above scenario Grafana will not send out notifications when other series cause the alert to fire if the rule already is in state `Alerting`. To improve support for queries that return multiple series we plan to track state **per series** in a future release.
|
||||
|
||||
> Starting with Grafana v5.3 you can configure reminders to be sent for triggered alerts. This will send additional notifications
|
||||
> when an alert continues to fire. If other series (like server2 in the example above) also cause the alert rule to fire they will be included in the reminder notification. Depending on what notification channel you're using you may be able to take advantage of this feature for identifying new/existing series causing alert to fire.
|
||||
|
||||
### No Data & Error Handling
|
||||
|
||||
Below are conditions you can configure how the rule evaluation engine should handle queries that return no data or only null values.
|
||||
|
||||
| No Data Option | Description |
|
||||
| --------------- | ------------------------------------------------------------------------------------------ |
|
||||
| No Data | Set alert rule state to `NoData` |
|
||||
| Alerting | Set alert rule state to `Alerting` |
|
||||
| Keep Last State | Keep the current alert rule state, whatever it is. |
|
||||
| Ok | Not sure why you would want to send yourself an alert when things are okay, but you could. |
|
||||
|
||||
### Execution errors or timeouts
|
||||
|
||||
Tell Grafana how to handle execution or timeout errors.
|
||||
|
||||
| Error or timeout option | Description |
|
||||
| ----------------------- | --------------------------------------------------- |
|
||||
| Alerting | Set alert rule state to `Alerting` |
|
||||
| Keep Last State | Keep the current alert rule state, whatever it is. |
|
||||
|
||||
If you have an unreliable time series store from which queries sometime timeout or fail randomly you can set this option to `Keep Last State` in order to basically ignore them.
|
||||
|
||||
## Notifications
|
||||
|
||||
In alert tab you can also specify alert rule notifications along with a detailed message about the alert rule. The message can contain anything, information about how you might solve the issue, link to runbook, and so on.
|
||||
|
||||
The actual notifications are configured and shared between multiple alerts. Read
|
||||
[Alert notifications]({{< relref "notifications.md" >}}) for information on how to configure and set up notifications.
|
||||
|
||||
- **Send to -** Select an alert notification channel if you have one set up.
|
||||
- **Message -** Enter a text message to be sent on the notification channel. Some alert notifiers support transforming the text to HTML or other rich formats. This field supports [templating]({{< relref "./add-notification-template.md" >}}).
|
||||
- **Tags -** Specify a list of tags (key/value) to be included in the notification. It is only supported by [some notifiers]({{< relref "notifications/#all-supported-notifiers" >}}).
|
||||
|
||||
## Alert state history and annotations
|
||||
|
||||
Alert state changes are recorded in the internal annotation table in Grafana's database. The state changes are visualized as annotations in the alert rule's graph panel. You can also go into the `State history` submenu in the alert tab to view and clear state history.
|
||||
|
||||
@@ -0,0 +1,283 @@
|
||||
+++
|
||||
title = "Alert notifications"
|
||||
description = "Alerting notifications guide"
|
||||
keywords = ["Grafana", "alerting", "guide", "notifications"]
|
||||
weight = 100
|
||||
+++
|
||||
|
||||
# Alert notifications
|
||||
|
||||
When an alert changes state, it sends out notifications. Each alert rule can have
|
||||
multiple notifications. In order to add a notification to an alert rule you first need
|
||||
to add and configure a `notification` channel (can be email, PagerDuty, or other integration).
|
||||
|
||||
This is done from the Notification channels page.
|
||||
|
||||
> **Note:** Alerting is only available in Grafana v4.0 and above.
|
||||
|
||||
## Add a notification channel
|
||||
|
||||
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Notification channels**.
|
||||
1. Click **Add channel**.
|
||||
1. Fill out the fields or select options described below.
|
||||
|
||||
## New notification channel fields
|
||||
|
||||
### Default (send on all alerts)
|
||||
|
||||
- **Name -** Enter a name for this channel. It will be displayed when users add notifications to alert rules.
|
||||
- **Type -** Select the channel type. Refer to the [List of supported notifiers](#list-of-supported-notifiers) for details.
|
||||
- **Default (send on all alerts) -** When selected, this option sends a notification on this channel for all alert rules.
|
||||
- **Include Image -** See [Enable images in notifications](#enable-images-in-notifications-external-image-store) for details.
|
||||
- **Disable Resolve Message -** When selected, this option disables the resolve message [OK] that is sent when the alerting state returns to false.
|
||||
- **Send reminders -** When this option is checked additional notifications (reminders) will be sent for triggered alerts. You can specify how often reminders should be sent using number of seconds (s), minutes (m) or hours (h), for example `30s`, `3m`, `5m` or `1h`.
|
||||
|
||||
**Important:** Alert reminders are sent after rules are evaluated. Therefore a reminder can never be sent more frequently than a configured alert rule evaluation interval.
|
||||
|
||||
These examples show how often and when reminders are sent for a triggered alert.
|
||||
|
||||
Alert rule evaluation interval | Send reminders every | Reminder sent every (after last alert notification)
|
||||
---------- | ----------- | -----------
|
||||
`30s` | `15s` | ~30 seconds
|
||||
`1m` | `5m` | ~5 minutes
|
||||
`5m` | `15m` | ~15 minutes
|
||||
`6m` | `20m` | ~24 minutes
|
||||
`1h` | `15m` | ~1 hour
|
||||
`1h` | `2h` | ~2 hours
|
||||
|
||||
<div class="clearfix"></div>
|
||||
|
||||
## List of supported notifiers
|
||||
|
||||
Name | Type | Supports images | Support alert rule tags
|
||||
-----|------|---------------- | -----------------------
|
||||
[DingDing](#dingdingdingtalk) | `dingding` | yes, external only | no
|
||||
[Discord](#discord) | `discord` | yes | no
|
||||
[Email](#email) | `email` | yes | no
|
||||
[Google Hangouts Chat](#google-hangouts-chat) | `googlechat` | yes, external only | no
|
||||
Hipchat | `hipchat` | yes, external only | no
|
||||
[Kafka](#kafka) | `kafka` | yes, external only | no
|
||||
Line | `line` | yes, external only | no
|
||||
Microsoft Teams | `teams` | yes, external only | no
|
||||
[Opsgenie](#opsgenie) | `opsgenie` | yes, external only | yes
|
||||
[Pagerduty](#pagerduty) | `pagerduty` | yes, external only | yes
|
||||
Prometheus Alertmanager | `prometheus-alertmanager` | yes, external only | yes
|
||||
[Pushover](#pushover) | `pushover` | yes | no
|
||||
Sensu | `sensu` | yes, external only | no
|
||||
[Sensu Go](#sensu-go) | `sensugo` | yes, external only | no
|
||||
[Slack](#slack) | `slack` | yes | no
|
||||
Telegram | `telegram` | yes | no
|
||||
Threema | `threema` | yes, external only | no
|
||||
VictorOps | `victorops` | yes, external only | yes
|
||||
[Webhook](#webhook) | `webhook` | yes, external only | yes
|
||||
[Zenduty](#zenduty) | `webhook` | yes, external only | yes
|
||||
|
||||
### Email
|
||||
|
||||
To enable email notifications you have to set up [SMTP settings]({{< relref "../../administration/configuration/#smtp" >}})
|
||||
in the Grafana config. Email notifications will upload an image of the alert graph to an
|
||||
external image destination if available or fallback to attaching the image to the email.
|
||||
Be aware that if you use the `local` image storage email servers and clients might not be
|
||||
able to access the image.
|
||||
|
||||
> **Note:** Template variables are not supported in email alerts.
|
||||
|
||||
Setting | Description
|
||||
---------- | -----------
|
||||
Single email | Send a single email to all recipients. Disabled per default.
|
||||
Addresses | Email addresses to recipients. You can enter multiple email addresses using a ";" separator.
|
||||
|
||||
### Slack
|
||||
|
||||
{{< figure class="float-right" max-width="40%" src="/static/img/docs/v4/slack_notification.png" caption="Alerting Slack Notification" >}}
|
||||
|
||||
To set up Slack, you need to configure an incoming Slack webhook URL. You can follow
|
||||
[Sending messages using Incoming Webhooks](https://api.slack.com/incoming-webhooks) on how to do that. If you want to include screenshots of the
|
||||
firing alerts in the Slack messages you have to configure either the [external image destination](#external-image-store)
|
||||
in Grafana or a bot integration via Slack Apps. [Follow Slack's guide to set up a bot integration](https://api.slack.com/bot-users) and use the token
|
||||
provided, which starts with "xoxb".
|
||||
|
||||
Setting | Description
|
||||
---------- | -----------
|
||||
Url | Slack incoming webhook URL, or eventually the [chat.postMessage](https://api.slack.com/methods/chat.postMessage) Slack API endpoint.
|
||||
Username | Set the username for the bot's message.
|
||||
Recipient | Allows you to override the Slack recipient. You must either provide a channel Slack ID, a user Slack ID, a username reference (@<user>, all lowercase, no whitespace), or a channel reference (#<channel>, all lowercase, no whitespace). If you use the `chat.postMessage` Slack API endpoint, this is required.
|
||||
Icon emoji | Provide an emoji to use as the icon for the bot's message. Ex :smile:
|
||||
Icon URL | Provide a URL to an image to use as the icon for the bot's message.
|
||||
Mention Users | Optionally mention one or more users in the Slack notification sent by Grafana. You have to refer to users, comma-separated, via their corresponding Slack IDs (which you can find by clicking the overflow button on each user's Slack profile).
|
||||
Mention Groups | Optionally mention one or more groups in the Slack notification sent by Grafana. You have to refer to groups, comma-separated, via their corresponding Slack IDs (which you can get from each group's Slack profile URL).
|
||||
Mention Channel | Optionally mention either all channel members or just active ones.
|
||||
Token | If provided, Grafana will upload the generated image via Slack's file.upload API method, not the external image destination. If you use the `chat.postMessage` Slack API endpoint, this is required.
|
||||
|
||||
If you are using the token for a slack bot, then you have to invite the bot to the channel you want to send notifications and add the channel to the recipient field.
|
||||
|
||||
### Opsgenie
|
||||
|
||||
To setup Opsgenie you will need an API Key and the Alert API Url. These can be obtained by configuring a new [Grafana Integration](https://docs.opsgenie.com/docs/grafana-integration).
|
||||
|
||||
Setting | Description
|
||||
--------|------------
|
||||
Alert API URL | The API URL for your Opsgenie instance. This will normally be either `https://api.opsgenie.com` or, for EU customers, `https://api.eu.opsgenie.com`.
|
||||
API Key | The API Key as provided by Opsgenie for your configured Grafana integration.
|
||||
Override priority | Configures the alert priority using the `og_priority` tag. The `og_priority` tag must have one of the following values: `P1`, `P2`, `P3`, `P4`, or `P5`. Default is `False`.
|
||||
Send notification tags as | Specify how you would like [Notification Tags]({{< relref "create-alerts.md/#notifications" >}}) delivered to Opsgenie. They can be delivered as `Tags`, `Extra Properties` or both. Default is Tags. See note below for more information.
|
||||
|
||||
> **Note:** When notification tags are sent as `Tags` they are concatenated into a string with a `key:value` format. If you prefer to receive the notifications tags as key/values under Extra Properties in Opsgenie then change the `Send notification tags as` to either `Extra Properties` or `Tags & Extra Properties`.
|
||||
|
||||
### PagerDuty
|
||||
|
||||
To set up PagerDuty, all you have to do is to provide an integration key.
|
||||
|
||||
Setting | Description
|
||||
---------- | -----------
|
||||
Integration Key | Integration key for PagerDuty.
|
||||
Severity | Level for dynamic notifications, default is `critical` (1)
|
||||
Auto resolve incidents | Resolve incidents in PagerDuty once the alert goes back to ok
|
||||
Message in details | Removes the Alert message from the PD summary field and puts it into custom details instead (2)
|
||||
|
||||
>**Note:** The tags `Severity`, `Class`, `Group`, `dedup_key`, and `Component` have special meaning in the [Pagerduty Common Event Format - PD-CEF](https://support.pagerduty.com/docs/pd-cef). If an alert panel defines these tag keys, then they are transposed to the root of the event sent to Pagerduty. This means they will be available within the Pagerduty UI and Filtering tools. A Severity tag set on an alert overrides the global Severity set on the notification channel if it's a valid level.
|
||||
|
||||
>Using Message In Details will change the structure of the `custom_details` field in the PagerDuty Event.
|
||||
This might break custom event rules in your PagerDuty rules if you rely on the fields in `payload.custom_details`.
|
||||
Move any existing rules using `custom_details.myMetric` to `custom_details.queries.myMetric`.
|
||||
This behavior will become the default in a future version of Grafana.
|
||||
|
||||
> **Note:** The `dedup_key` tag overrides the Grafana-generated `dedup_key` with a custom key.
|
||||
|
||||
> **Note:** The `state` tag overrides the current alert state inside the `custom_details` payload.
|
||||
|
||||
### VictorOps
|
||||
|
||||
To configure VictorOps, provide the URL from the Grafana Integration and substitute `$routing_key` with a valid key.
|
||||
|
||||
> **Note:** The tag `Severity` has special meaning in the [VictorOps Incident Fields](https://help.victorops.com/knowledge-base/incident-fields-glossary/). If an alert panel defines this key, then it replaces the `message_type` in the root of the event sent to VictorOps.
|
||||
### Pushover
|
||||
|
||||
To set up Pushover, you must provide a user key and an API token. Refer to [What is Pushover and how do I use it](https://support.pushover.net/i7-what-is-pushover-and-how-do-i-use-it) for instructions on how to generate them.
|
||||
|
||||
|
||||
Setting | Description
|
||||
---------- | -----------
|
||||
API Token | Application token
|
||||
User key(s) | A comma-separated list of user keys
|
||||
Device(s) | A comma-separated list of devices
|
||||
Priority | The priority alerting nottifications are sent
|
||||
OK priority | The priority OK notifications are sent; if not set, then OK notifications are sent with the priority set for alerting notifications
|
||||
Retry | How often (in seconds) the Pushover servers send the same notification to the user. (minimum 30 seconds)
|
||||
Expire | How many seconds your notification will continue to be retried for (maximum 86400 seconds)
|
||||
Alerting sound | The sound for alerting notifications
|
||||
OK sound | The sound for OK notifications
|
||||
|
||||
### Webhook
|
||||
|
||||
The webhook notification is a simple way to send information about a state change over HTTP to a custom endpoint.
|
||||
Using this notification you could integrate Grafana into a system of your choosing.
|
||||
|
||||
Example json body:
|
||||
|
||||
```json
|
||||
{
|
||||
"dashboardId":1,
|
||||
"evalMatches":[
|
||||
{
|
||||
"value":1,
|
||||
"metric":"Count",
|
||||
"tags":{}
|
||||
}
|
||||
],
|
||||
"imageUrl":"https://grafana.com/assets/img/blog/mixed_styles.png",
|
||||
"message":"Notification Message",
|
||||
"orgId":1,
|
||||
"panelId":2,
|
||||
"ruleId":1,
|
||||
"ruleName":"Panel Title alert",
|
||||
"ruleUrl":"http://localhost:3000/d/hZ7BuVbWz/test-dashboard?fullscreen\u0026edit\u0026tab=alert\u0026panelId=2\u0026orgId=1",
|
||||
"state":"alerting",
|
||||
"tags":{
|
||||
"tag name":"tag value"
|
||||
},
|
||||
"title":"[Alerting] Panel Title alert"
|
||||
}
|
||||
```
|
||||
|
||||
- **state** - The possible values for alert state are: `ok`, `paused`, `alerting`, `pending`, `no_data`.
|
||||
|
||||
### DingDing/DingTalk
|
||||
|
||||
DingTalk supports the following "message type": `text`, `link` and `markdown`. Only the `link` message type is supported. Refer to the [configuration instructions](https://developers.dingtalk.com/document/app/custom-robot-access) in Chinese language.
|
||||
|
||||
In DingTalk PC Client:
|
||||
|
||||
1. Click "more" icon on upper right of the panel.
|
||||
|
||||
2. Click "Robot Manage" item in the pop menu, there will be a new panel call "Robot Manage".
|
||||
|
||||
3. In the "Robot Manage" panel, select "customized: customized robot with Webhook".
|
||||
|
||||
4. In the next new panel named "robot detail", click "Add" button.
|
||||
|
||||
5. In "Add Robot" panel, input a nickname for the robot and select a "message group" which the robot will join in. click "next".
|
||||
|
||||
6. There will be a Webhook URL in the panel, looks like this: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxx. Copy this URL to the Grafana DingTalk setting page and then click "finish".
|
||||
|
||||
### Discord
|
||||
|
||||
To set up Discord, you must create a Discord channel webhook. For instructions on how to create the channel, refer to
|
||||
[Intro to Webhooks](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks) f.
|
||||
|
||||
Setting | Description
|
||||
---------- | -----------
|
||||
Webhook URL | Discord webhook URL.
|
||||
Message Content | Mention a group using @ or a user using <@ID> when notifying in a channel.
|
||||
Avatar URL | Optionally, provide a URL to an image to use as the avatar for the bot's message.
|
||||
|
||||
Alternately, use the [Slack](#slack) notifier by appending `/slack` to a Discord webhook URL.
|
||||
|
||||
### Kafka
|
||||
|
||||
Notifications can be sent to a Kafka topic from Grafana using the [Kafka REST Proxy](https://docs.confluent.io/1.0/kafka-rest/docs/index.html).
|
||||
There are a couple of configuration options which need to be set up in Grafana UI under Kafka Settings:
|
||||
|
||||
1. Kafka REST Proxy endpoint.
|
||||
|
||||
1. Kafka Topic.
|
||||
|
||||
Once these two properties are set, you can send the alerts to Kafka for further processing or throttling.
|
||||
|
||||
### Google Hangouts Chat
|
||||
|
||||
Notifications can be sent by setting up an incoming webhook in Google Hangouts chat. For more information about configuring a webhook, refer to [webhooks](https://developers.google.com/hangouts/chat/how-tos/webhooks).
|
||||
|
||||
### Prometheus Alertmanager
|
||||
|
||||
Alertmanager handles alerts sent by client applications such as Prometheus server or Grafana. It takes care of deduplicating, grouping, and routing them to the correct receiver. Grafana notifications can be sent to Alertmanager via a simple incoming webhook. Refer to the official [Prometheus Alertmanager documentation](https://prometheus.io/docs/alerting/alertmanager) for configuration information.
|
||||
|
||||
> **Caution:** In case of a high-availability setup, do not load balance traffic between Grafana and Alertmanagers to keep coherence between all your Alertmanager instances. Instead, point Grafana to a list of all Alertmanagers, by listing their URLs comma-separated in the notification channel configuration.
|
||||
|
||||
### Zenduty
|
||||
|
||||
[Zenduty](https://www.zenduty.com) is an incident alerting and response orchestration platform that not alerts the right teams via SMS, Phone(Voice), Email, Slack, Microsoft Teams and Push notifications(Android/iOS) whenever a Grafana alert is triggered, but also helps you rapidly triage and remediate critical, user impacting incidents. Grafana alert are sent to Zenduty through Grafana's native webhook dispatcher. Refer the Zenduty-Grafana [integration documentation](https://docs.zenduty.com/docs/grafana) for configuring the integration.
|
||||
|
||||
### Sensu Go
|
||||
|
||||
[Sensu](https://sensu.io) is a complete solution for monitoring and observability at scale. Sensu Go is designed to give you visibility into everything you care about: traditional server closets, containers, applications, the cloud, and more. Grafana notifications can be sent to Sensu Go as events via the API. This operation requires an API Key. Refer to the [Sensu Go documentation](https://docs.sensu.io/sensu-go/latest/operations/control-access/use-apikeys/#api-key-authentication) for information on creating this key.
|
||||
|
||||
## Enable images in notifications {#external-image-store}
|
||||
|
||||
Grafana can render the panel associated with the alert rule as a PNG image and include that in the notification. Read more about the requirements and how to configure
|
||||
[image rendering]({{< relref "../../administration/image_rendering/" >}}).
|
||||
|
||||
You must configure an [external image storage provider]({{< relref "../../administration/configuration/#external-image-storage" >}}) in order to receive images in alert notifications. If your notification channel requires that the image be publicly accessible (e.g. Slack, PagerDuty), configure a provider which uploads the image to a remote image store like Amazon S3, Webdav, Google Cloud Storage, or Azure Blob Storage. Otherwise, the local provider can be used to serve the image directly from Grafana.
|
||||
|
||||
Notification services which need public image access are marked as 'external only'.
|
||||
|
||||
## Configure the link back to Grafana from alert notifications
|
||||
|
||||
All alert notifications contain a link back to the triggered alert in the Grafana instance.
|
||||
This URL is based on the [domain]({{< relref "../../administration/configuration/#domain" >}}) setting in Grafana.
|
||||
|
||||
## Notification templating
|
||||
|
||||
> **Note:** Alert notification templating is only available in Grafana v7.4 and above.
|
||||
|
||||
The alert notification template feature allows you to take the [label]({{< relref "../../basics/timeseries-dimensions.md#labels" >}}) value from an alert query and [inject that into alert notifications]({{< relref "./add-notification-template.md" >}}).
|
||||
@@ -0,0 +1,14 @@
|
||||
+++
|
||||
title = "Pause alert rule"
|
||||
description = "Pause an existing alert rule"
|
||||
keywords = ["grafana", "alerting", "guide", "rules", "view"]
|
||||
weight = 400
|
||||
+++
|
||||
|
||||
# Pause an alert rule
|
||||
|
||||
Pausing the evaluation of an alert rule can sometimes be useful. For example, during a maintenance window, pausing alert rules can avoid triggering a flood of alerts.
|
||||
|
||||
1. In the Grafana side bar, hover your cursor over the Alerting (bell) icon and then click **Alert Rules**. All configured alert rules are listed, along with their current state.
|
||||
1. Find your alert in the list, and click the **Pause** icon on the right. The **Pause** icon turns into a **Play** icon.
|
||||
1. Click the **Play** icon to resume evaluation of your alert.
|
||||
@@ -0,0 +1,41 @@
|
||||
+++
|
||||
title = "Troubleshoot alerts"
|
||||
description = "Troubleshoot alert rules"
|
||||
keywords = ["grafana", "alerting", "guide", "rules", "troubleshoot"]
|
||||
weight = 500
|
||||
+++
|
||||
|
||||
# Troubleshoot alerts
|
||||
|
||||
If alerts are not behaving as you expect, here are some steps you can take to troubleshoot and figure out what is going wrong.
|
||||
|
||||
|
||||
|
||||
The first level of troubleshooting you can do is click **Test Rule**. You will get result back that you can expand to the point where you can see the raw data that was returned from your query.
|
||||
|
||||
Further troubleshooting can also be done by inspecting the grafana-server log. If it's not an error or for some reason the log does not say anything you can enable debug logging for some relevant components. This is done in Grafana's ini config file.
|
||||
|
||||
Example showing loggers that could be relevant when troubleshooting alerting.
|
||||
|
||||
```ini
|
||||
[log]
|
||||
filters = alerting.scheduler:debug \
|
||||
alerting.engine:debug \
|
||||
alerting.resultHandler:debug \
|
||||
alerting.evalHandler:debug \
|
||||
alerting.evalContext:debug \
|
||||
alerting.extractor:debug \
|
||||
alerting.notifier:debug \
|
||||
alerting.notifier.slack:debug \
|
||||
alerting.notifier.pagerduty:debug \
|
||||
alerting.notifier.email:debug \
|
||||
alerting.notifier.webhook:debug \
|
||||
tsdb.graphite:debug \
|
||||
tsdb.prometheus:debug \
|
||||
tsdb.opentsdb:debug \
|
||||
tsdb.influxdb:debug \
|
||||
tsdb.elasticsearch:debug \
|
||||
tsdb.elasticsearch.client:debug \
|
||||
```
|
||||
|
||||
If you want to log raw query sent to your TSDB and raw response in log you also have to set grafana.ini option `app_mode` to `development`.
|
||||
@@ -0,0 +1,19 @@
|
||||
+++
|
||||
title = "View alerts"
|
||||
description = "View existing alert rules"
|
||||
keywords = ["grafana", "alerting", "guide", "rules", "view"]
|
||||
weight = 400
|
||||
+++
|
||||
|
||||
# View existing alert rules
|
||||
|
||||
Grafana stores individual alert rules in the panels where they are defined, but you can also view a list of all existing alert rules and their current state.
|
||||
|
||||
In the Grafana side bar, hover your cursor over the Alerting (bell) icon and then click **Alert Rules**. All configured alert rules are listed, along with their current state.
|
||||
|
||||
You can do several things while viewing alerts.
|
||||
|
||||
- **Filter alerts by name -** Type an alert name in the **Search alerts** field.
|
||||
- **Filter alerts by state -** In **States**, select which alert states you want to see. All others will be hidden.
|
||||
- **Pause or resume an alert -** Click the **Pause** or **Play** icon next to the alert to pause or resume evaluation. See [Pause an alert rule]({{< relref "pause-an-alert-rule.md" >}}) for more information.
|
||||
- **Access alert rule settings -** Click the alert name or the **Edit alert rule** (gear) icon. Grafana opens the Alert tab of the panel where the alert rule is defined. This is helpful when an alert is firing but you don't know which panel it is defined in.
|
||||
@@ -0,0 +1,60 @@
|
||||
+++
|
||||
title = "Grafana 8 Alerts"
|
||||
aliases = ["/docs/grafana/latest/alerting/metrics/"]
|
||||
weight = 113
|
||||
+++
|
||||
|
||||
# Grafana 8 alerts
|
||||
Alerts allow you to know about problems in your systems moments after they occur. Robust and actionable alerts help you identify and resolve issues quickly, minimizing disruption to your services.
|
||||
|
||||
>**Note:** This information is for the new, Grafana 8 Alerts. This is an [opt-in]({{< relref"./opt-in.md" >}}) feature released in Grafana 8.0. Grafana still supports [dashboard alerts]({{< relref "../old-alerting/_index.md" >}}) out of the box
|
||||
|
||||
Alerts have four main components:
|
||||
|
||||
- Alerting rule - One or more query and/or expression, a condition, the frequency of evaluation, and the (optional) duration that a condition must be met before creating an alert.
|
||||
- Contact point - A channel for sending notifications when the conditions of an alerting rule are met.
|
||||
- Notification policy - A set of matching and grouping criteria used to determine where, and how frequently, to send notifications.
|
||||
- Silences - Date and matching criteria used to silence notifications.
|
||||
|
||||
## Alerting tasks
|
||||
|
||||
You can perform the following tasks for alerts:
|
||||
|
||||
|
||||
- [Create a Grafana managed alert rule]({{< relref "alerting-rules/create-grafana-managed-rule.md" >}})
|
||||
- [Create a Cortex or Loki managed alert rule]({{< relref "alerting-rules/create-cortex-loki-managed-rule.md" >}})
|
||||
- [View existing alert rules and their current state]({{< relref "alerting-rules/rule-list.md" >}})
|
||||
- [Test alert rules and troubleshoot]({{< relref "./troubleshoot-alerts.md" >}})
|
||||
- [Add or edit an alert contact point]({{< relref "./contact-points.md" >}})
|
||||
- [Add or edit notification policies]({{< relref "./notification-policies.md" >}})
|
||||
- [Create and edit silences]({{< relref "./silences.md" >}})
|
||||
|
||||
## Clustering
|
||||
|
||||
Currently alerting supports a limited form of high availability. Alert notifications are deduped when running multiple servers. This means all alerts are executed on every server but no duplicate alert notifications are sent due to the deduping logic. Proper load balancing of alerts will be introduced in the future.
|
||||
|
||||
## Alert evaluation
|
||||
|
||||
Grafana managed alerts are evaluated by the Grafana backend. Rule evaluations are scheduled, according to the alert rule configuration, and queries are evaluated by an engine that is part of core Grafana.
|
||||
|
||||
Alerting rules can only query backend data sources with alerting enabled:
|
||||
- builtin or developed and maintained by grafana: `Graphite`, `Prometheus`, `Loki`, `InfluxDB`, `Elasticsearch`,
|
||||
`Google Cloud Monitoring`, `Cloudwatch`, `Azure Monitor`, `MySQL`, `PostgreSQL`, `MSSQL`, `OpenTSDB`, `Oracle`, and `Azure Data Explorer`
|
||||
- any community backend data sources with alerting enabled (`backend` and `alerting` properties are set in the [plugin.json]({{< relref "../../developers/plugins/metadata.md" >}}))
|
||||
|
||||
## Metrics from the alerting engine
|
||||
|
||||
The alerting engine publishes some internal metrics about itself. You can read more about how Grafana publishes [internal metrics]({{< relref "../../administration/view-server/internal-metrics.md" >}}).
|
||||
|
||||
Metric Name | Type | Description
|
||||
---------- | ----------- | ----------
|
||||
`alerting.alerts` | gauge | How many alerts by state
|
||||
`alerting.request_duration_seconds` | histogram | Histogram of requests to the Alerting API
|
||||
`alerting.active_configurations` | gauge | The number of active, non default Alertmanager configurations for grafana managed alerts
|
||||
`alerting.rule_evaluations_total` | counter | The total number of rule evaluations
|
||||
`alerting.rule_evaluation_failures_total` | counter | The total number of rule evaluation failures
|
||||
`alerting.rule_evaluation_duration_seconds` | summary | The duration for a rule to execute
|
||||
`alerting.rule_group_rules` | gauge | The number of rules
|
||||
|
||||
|
||||
- [View alert rules and their current state]({{< relref "alerting-rules/rule-list.md" >}})
|
||||
@@ -0,0 +1,12 @@
|
||||
+++
|
||||
title = "Alerting Rules"
|
||||
aliases = ["/docs/grafana/latest/alerting/rules/"]
|
||||
+++
|
||||
|
||||
# Alerting Rules
|
||||
One or more queries and/or expressions, a condition, the frequency of evaluation, and the (optional) duration that a condition must be met before creating an alert. Alerting rules are how you express the criteria for creating an alert. Queries and expressions select and can operate on the data you wish to alert on. A condition sets the threshold that an alert must meet or exceed to create an alert. The interval specifies how frequently the rule should be evaluated. The duration, when configured, sets a period that a condition must be met or exceeded before an alert is created. Alerting rules also can contain settings for what to do when your query does not return any data, or there is an error attempting to execute the query.
|
||||
|
||||
- [View existing alert rules and their current state]({{< relref "./rule-list.md" >}})
|
||||
- [Create Cortex or Loki managed alert rule]({{< relref "./create-cortex-loki-managed-rule.md" >}})
|
||||
- [Create Grafana managed alert rule]({{< relref "./create-grafana-managed-rule.md" >}})
|
||||
- [State and Health of alerting rules]({{< relref "./state-and-health.md" >}})
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
+++
|
||||
title = "Create Cortex or Loki managed alert rule"
|
||||
description = "Create Cortex or Loki managed alerting rule"
|
||||
keywords = ["grafana", "alerting", "guide", "rules", "create"]
|
||||
weight = 400
|
||||
+++
|
||||
|
||||
# Create a Cortex or Loki managed alerting rule
|
||||
|
||||
Grafana allows you manage alerting rules for an external Cortex or Loki instance.
|
||||
|
||||
## Add or edit a Cortex or Loki managed alerting rule
|
||||
|
||||
1. In the Grafana menu hover your cursor over the Alerting (bell) icon.
|
||||
1. To create a new alert rule, click **New alert rule**. To edit an existing rule, expand one of the rules in the **Cortex / Loki** section and click **Edit**.
|
||||
1. Click on the **Alert type** drop down and select **Cortex / Loki managed alert**.
|
||||
1. Fill out the rest of the fields. Descriptions are listed below in [Alert rule fields](#alert-rule-fields).
|
||||
1. When you have finished writing your rule, click **Save** in the upper right corner to save the rule,, or **Save and exit** to save and exit rule editing.
|
||||
|
||||
## Alert rule fields
|
||||
|
||||
This section describes the fields you fill out to create an alert.
|
||||
|
||||
### Alert type
|
||||
|
||||
- **Alert name -** Enter a descriptive name. The name will be displayed in the alert rule list, as well as added as `alertname` label to every alert instance that is created from this rule.
|
||||
- **Alert type -** Select **Cortex / Loki managed alert**.
|
||||
- **Data source -** Select a Prometheus or Loki data source. Only Prometheus data sources that support Cortex ruler API will be available.
|
||||
- **Namespace -** Select an existing rule namespace or click **Add new** to create a new one.
|
||||
- **Group -** Select an existing group within the selected namespace or click **Add new** to create a new one. Newly created rules will be added to the end of the rule group.
|
||||
|
||||
|
||||
|
||||
### Query
|
||||
|
||||
Enter a PromQL or LogQL expression. Rule will fire if evaluation result has at least one series with value > 0. An alert will be created per each such series.
|
||||
|
||||
|
||||
|
||||
### Conditions
|
||||
|
||||
- **For -** For how long the selected condition should violated before an alert enters `Firing` state. When condition threshold is violated for the first time, an alert becomes `Pending`. If the **for** time elapses and the condition is still violated, it becomes `Firing`. Else it reverts back to `Normal`.
|
||||
|
||||
|
||||
|
||||
### Details
|
||||
|
||||
Annotations and labels can be optionally added in the details section.
|
||||
|
||||
#### Annotations
|
||||
|
||||
Annotations are key and value pairs that provide additional meta information about the alert, for example description, summary, runbook URL. They are displayed in rule and alert details in the UI and can be used in contact type message templates. Annotations can also be templated, for example `Instance {{ $labels.instance }} down` will have the evaluated `instance` label value added for every alert this rule produces.
|
||||
|
||||
#### Labels
|
||||
|
||||
Labels are key value pairs that categorize or identify an alert. Labels are used to match alerts in silences or match and groups alerts in notification policies. Labels are also shown in rule or alert details in the UI and can be used in contact type message templates. For example, it is common to add a `severity` label and then configure a separate notification policy for each severity. Or one could add a `team` label and configure team specific notification policies, or silence all alerts for a particular team.
|
||||
|
||||
|
||||
|
||||
## Preview alerts
|
||||
|
||||
To evaluate the rule and see what alerts it would produce, click **Preview alerts**. It will display a list of alerts with state and value of for each one.
|
||||
@@ -0,0 +1,108 @@
|
||||
+++
|
||||
title = "Create Grafana managed alert rule"
|
||||
description = "Create Grafana managed alert rule"
|
||||
keywords = ["grafana", "alerting", "guide", "rules", "create"]
|
||||
weight = 400
|
||||
+++
|
||||
|
||||
# Create a Grafana managed alerting rule
|
||||
|
||||
Grafana allows you to create alerting rules that query one or more data sources, reduce or transform the results and compare them to each other or to fix thresholds. These rules will be executed and notifications sent by Grafana itself.
|
||||
|
||||
## Add or edit a Grafana managed alerting rule
|
||||
|
||||
1. In the Grafana menu hover your cursor over the Alerting (bell) icon.
|
||||
1. To create a new alert rule, click **New alert rule**. To edit an existing rule, expand one of the rules in the **Grafana** section and click **Edit**.
|
||||
1. Click on the **Alert type** drop down and select **Grafana managed alert**.
|
||||
1. Fill out the rest of the fields. Descriptions are listed below in [Alert rule fields](#alert-rule-fields).
|
||||
1. When you have finished writing your rule, click **Save** in the upper right corner to save the rule,, or **Save and exit** to save and exit rule editing.
|
||||
|
||||
## Alert rule fields
|
||||
|
||||
This section describes the fields you fill out to create an alert.
|
||||
|
||||
### Alert type
|
||||
|
||||
- **Alert name -** Enter a descriptive name. The name will be displayed in the alert rule list, as well as added as `alertname` label to every alert instance that is created from this rule.
|
||||
- **Alert type -** Select **Grafana managed alert**.
|
||||
- **Folder -** Select a folder this alert rule will belong to. To create a new folder, click on the drop down and type in a new folder name.
|
||||
|
||||
|
||||
|
||||
### Query
|
||||
|
||||
Add one or more [queries]({{< relref "../../../panels/queries.md" >}}) or [expressions]({{< relref "../../../panels/expressions.md" >}}). You can use classic condition expression to create a rule that will trigger a single alert if it's threshold is met, or use reduce and math expressions to create a multi dimensional alert rule that can trigger multiple alerts, one per matching series in the query result.
|
||||
|
||||
#### Rule with classic condition
|
||||
|
||||
You can use classic condition expression to create a rule that will trigger a single alert if it's conditions is met. It works about the same way as dashboard alerts in previous versions of Grafana.
|
||||
|
||||
1. Add one or more queries
|
||||
1. Add an expression. Click on **Operation** dropdown and select **Classic condition**.
|
||||
1. Add one or more conditions. For each condition you can specify operator (`AND` / `OR`), aggregation function, query letter and threshold value.
|
||||
|
||||
If a query returns multiple series, then the aggregation function and threshold check will be evaluated for each series.It will not track alert state **per series**. This has implications that are detailed in the scenario below.
|
||||
|
||||
- Alert condition with query that returns 2 series: **server1** and **server2**
|
||||
- **server1** series causes the alert rule to fire and switch to state `Firing`
|
||||
- Notifications are sent out with message: _load peaking (server1)_
|
||||
- In a subsequent evaluation of the same alert rule, the **server2** series also causes the alert rule to fire
|
||||
- No new notifications are sent as the alert rule is already in state `Firing`.
|
||||
|
||||
So, as you can see from the above scenario Grafana will not send out notifications when other series cause the alert to fire if the rule already is in state `Firing`. If you want to have alert per series, create a multi dimensional alert rule as described in the section below.
|
||||
|
||||
|
||||
|
||||
#### Multi dimensional rule
|
||||
|
||||
You can use reduce and math expressions to create a rule that will create an alert per series returned by the query.
|
||||
|
||||
1. Add one or more queries
|
||||
2. Add a `reduce` expression for each query to aggregate values in the selected time range into a single value. With some data sources this is not needed for [rules using numeric data]({{< relref "../grafana-managed-numeric-rule.md" >}}).
|
||||
3. Add a `math` expressions with the condition for the rule. Not needed in case a query or a reduce expression already returns 0 if rule should not be firing, or > 0 if it should be firing. Some examples: `$B > 70` if it should fire in case value of B query/expression is more than 70. `$B < $C * 100` in case it should fire if value of B is less than value of C multiplied by 100. If queries being compared have multiple series in their results, series from different queries are matched if they have the same labels or one is a subset of the other.
|
||||
|
||||
See or [expressions documentation]({{< relref "../../../panels/expressions.md" >}}) for in depth explanation of `math` and `reduce` expressions.
|
||||
|
||||
|
||||
|
||||
### Conditions
|
||||
|
||||
- **Condition -** Select the letter of the query or expression whose result will trigger the alert rule. You will likely want to select either a `classic condition` or a `math` expression.
|
||||
- **Evaluate every -** How often the rule should be evaluated, executing the defined queries and expressions. Must be no less than 10 seconds and a multiple of 10 seconds. Examples: `1m`, `30s`
|
||||
- **Evaluate for -** For how long the selected condition should violated before an alert enters `Alerting` state. When condition threshold is violated for the first time, an alert becomes `Pending`. If the **for** time elapses and the condition is still violated, it becomes `Alerting`. Else it reverts back to `Normal`.
|
||||
|
||||
#### No Data & Error handling
|
||||
|
||||
Toggle **Configure no data and error handling** switch to configure how the rule should handle cases where evaluation results in error or returns no data.
|
||||
|
||||
| No Data Option | Description |
|
||||
| --------------- | ------------------------------------------------------------------------------------------ |
|
||||
| No Data | Set alert state to `NoData` and rule state to `Normal` |
|
||||
| Alerting | Set alert rule state to `Alerting` |
|
||||
| Ok | Set alert rule state to `Normal` |
|
||||
|
||||
|
||||
| Error or timeout option | Description |
|
||||
| ----------------------- | --------------------------------------------------- |
|
||||
| Alerting | Set alert rule state to `Alerting` |
|
||||
| OK | Set alert rule state to `Normal` |
|
||||
|
||||
|
||||
|
||||
### Details
|
||||
|
||||
Annotations and labels can be optionally added in the details section.
|
||||
|
||||
#### Annotations
|
||||
|
||||
Annotations are key and value pairs that provide additional meta information about the alert, for example description, summary, runbook URL. They are displayed in rule and alert details in the UI and can be used in contact type message templates. Annotations can also be templated, for example `Instance {{ $labels.instance }} down` will have the evaluated `instance` label value added for every alert this rule produces.
|
||||
|
||||
#### Labels
|
||||
|
||||
Labels are key value pairs that categorize or identify an alert. Labels are used to match alerts in silences or match and groups alerts in notification policies. Labels are also shown in rule or alert details in the UI and can be used in contact type message templates. For example, it is common to add a `severity` label and then configure a separate notification policy for each severity. Or one could add a `team` label and configure team specific notification policies, or silence all alerts for a particular team. Labels can also be templated like annotations, for example `{{ $labels.namespace }}/{{ $labels.job }}` will produce a new rule label that will have the evaluated `namespace` and `job` label value added for every alert this rule produces. The rule labels take precedence over the labels produced by the query/condition.
|
||||
|
||||
|
||||
|
||||
## Preview alerts
|
||||
|
||||
To evaluate the rule and see what alerts it would produce, click **Preview alerts**. It will display a list of alerts with state and value for each one.
|
||||
@@ -0,0 +1,48 @@
|
||||
+++
|
||||
title = "View alert rules"
|
||||
description = "View alert rules"
|
||||
keywords = ["grafana", "alerting", "guide", "rules", "view"]
|
||||
weight = 400
|
||||
+++
|
||||
|
||||
# View alert rules
|
||||
|
||||
To view alters:
|
||||
1. In the Grafana menu hover your cursor over the Alerting (bell) icon.
|
||||
1. Click **Alert Rules**. You can see all configured Grafana alert rules as well as any rules from Loki or Prometheus data sources.
|
||||
By default, the group view is shown. You can toggle between group or state views by clicking the relevant **View as** buttons in the options area at the top of the page.
|
||||
|
||||
### Group view
|
||||
|
||||
Group view shows Grafana alert rules grouped by folder and Loki or Prometheus alert rules grouped by `namespace` + `group`. This is the default rule list view, intended for managing rules. You can expand each group to view a list of rules in this group. Each rule can be further expanded to view its details. Action buttons and any alerts spawned by this rule, and each alert can be further expanded to view its details.
|
||||
|
||||
|
||||
|
||||
### State view
|
||||
|
||||
State view shows alert rules grouped by state. Use this view to get an overview of which rules are in what state. Each rule can be expanded to view its details. Action buttons and any alerts spawned by this rule, and each alert can be further expanded to view its details.
|
||||
|
||||
|
||||
|
||||
## Filter alert rules
|
||||
You can use the following filters to view only alert rules that match specific criteria:
|
||||
|
||||
- **Filter alerts by name or label -** Type an alert name, label name or value in the **Search** input.
|
||||
- **Filter alerts by state -** In **States** Select which alert states you want to see. All others are hidden.
|
||||
- **Filter alerts by data source -** Click the **Select data source** and select an alerting data source. Only alert rules that query selected data source will be visible.
|
||||
|
||||
## Rule details
|
||||
|
||||
A rule row shows the rule state, health, and summary annotation if the rule has one. You can expand the rule row to display rule labels, all annotations, data sources this rule queries, and a list of alert instances spawned from this rule.
|
||||
|
||||
|
||||
|
||||
### Edit or delete rule
|
||||
|
||||
Grafana rules can only be edited or deleted by users with Edit permissions for the folder which contains the rule. Prometheus or Loki rules can be edited or deleted by users with Editor or Admin roles.
|
||||
|
||||
To edit or delete a rule:
|
||||
|
||||
1. Expand this rule to reveal rule controls.
|
||||
1. Click **Edit** to go to the rule editing form. Make changes following [instructions listed here]({{< relref "./create-grafana-managed-rule.md" >}}).
|
||||
1. Click **Delete"** to delete a rule.
|
||||
@@ -0,0 +1,27 @@
|
||||
+++
|
||||
title = "State and Health of alerting rules"
|
||||
description = "State and Health of alerting rules"
|
||||
keywords = ["grafana", "alerting", "guide", "state"]
|
||||
+++
|
||||
|
||||
The concepts of state and health for alerting rules help you understand, at a glance, several key status indicators about your alerts. Alert state, alerting rule state, and alerting rule health are related, but they each convey subtly different information.
|
||||
|
||||
## Alerting rule state
|
||||
Indicates whether any of the timeseries resulting from evaluation of the alerting rule are in an alerting state. Alerting rule state only requires a single alerting instance to be in a pending or firing state for the alerting rule state to not be normal.
|
||||
- Normal: none of the timeseries returned are in an alerting state.
|
||||
- Pending: at least one of the timeseries returned are in a pending state.
|
||||
- Firing: at least one of the timeseries returned are in an alerting state.
|
||||
|
||||
## Alert state
|
||||
Alert state is an indication of the output of the alerting evaluation engine.
|
||||
- Normal: the condition for the alerting rule has evaluated to **false** for every timeseries returned by the evaluation engine.
|
||||
- Alerting: the condition for the alerting rule has evaluated to **true** for at least one timeseries returned by the evaluation engine and the duration, if set, **has** been met or exceeded.
|
||||
- Pending: the condition for the alerting rule has evaluated to **true** for at least one timeseries returned by the evaluation engine and the duration, if set, **has not** been met or exceeded.
|
||||
- NoData: the alerting rule has not returned a timeseries, all values for the timeseries are null, or all values for the timeseries are zero.
|
||||
- Error: There was an error encountered when attempting to evaluate the alerting rule.
|
||||
|
||||
## Alerting rule health
|
||||
Indicates the status of alerting rule evaluation.
|
||||
- Ok: the rule is being evaluated, data is being returned, and no errors have been encountered.
|
||||
- Error: an error was encountered when evaluating the alerting rule.
|
||||
- NoData: at least one of the timeseries returned during evaluation is in a NoData state.
|
||||
@@ -0,0 +1,73 @@
|
||||
+++
|
||||
title = "Contact points"
|
||||
description = "Create or edit contact point"
|
||||
keywords = ["grafana", "alerting", "guide", "contact point", "notification channel", "create"]
|
||||
weight = 400
|
||||
+++
|
||||
|
||||
# Contact points
|
||||
|
||||
Contact points define where to send notifications about alerts that match a particular [notification policy]({{< relref "./notification-policies.md" >}}). A contact point can contain one or more contact point types, eg email, slack, webhook and so on. A notification will dispatched to all contact point types defined on a contact point. [Templating]({{< relref "./message-templating/_index.md" >}}) can be used to customize contact point type message with alert data. Grafana alerting UI can be used to configure both Grafana managed contact points and contact points for an [external Alertmanager if one is configured]({{< relref "../../datasources/alertmanager.md" >}}).
|
||||
|
||||
## Add a contact point
|
||||
|
||||
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
|
||||
1. Click **Add contact point**.
|
||||
1. Enter a **Name** for the contact point
|
||||
1. Select contact point type and fill out mandatory fields. **Optional settings** can be expanded for more options.
|
||||
1. If you'd like this contact point to notify via multiple channels, for example both email and slack, click **New contact point type** and fill out additional contact point type details.
|
||||
1. Click **Save contact point** button at the bottom of the page.
|
||||
|
||||
## Editing a contact point
|
||||
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
|
||||
1. Find the contact point you want to edit in the contact points table and click the **pen icon** on the right side.
|
||||
1. Make any changes and click **Save contact point** button at the bottom of the page.
|
||||
|
||||
## Deleting a contact point
|
||||
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
|
||||
1. Find the contact point you want to edit in the contact points table and click the **trash can icon** on the right side.
|
||||
1. A confirmation dialog will open. Click **Yes, delete**.
|
||||
|
||||
**Note** You will not be able to delete contact points that are currently used by any notification policy. If you want to delete such contact point, you will have to first go to [notification policies]({{< relref "./notification-policies.md" >}}) and delete the policy or update it to use another contact point.
|
||||
|
||||
## List of notifiers supported by Grafana
|
||||
|
||||
Name | Type
|
||||
-----|-----
|
||||
[DingDing](#dingdingdingtalk) | `dingding`
|
||||
[Discord](#discord) | `discord`
|
||||
[Email](#email) | `email`
|
||||
[Google Hangouts Chat](#google-hangouts-chat) | `googlechat`
|
||||
[Kafka](#kafka) | `kafka`
|
||||
Line | `line`
|
||||
Microsoft Teams | `teams`
|
||||
[Opsgenie](#opsgenie) | `opsgenie`
|
||||
[Pagerduty](#pagerduty) | `pagerduty`
|
||||
Prometheus Alertmanager | `prometheus-alertmanager`
|
||||
[Pushover](#pushover) | `pushover`
|
||||
Sensu | `sensu`
|
||||
[Sensu Go](#sensu-go) | `sensugo`
|
||||
[Slack](#slack) | `slack`
|
||||
Telegram | `telegram`
|
||||
Threema | `threema`
|
||||
VictorOps | `victorops`
|
||||
[Webhook](#webhook) | `webhook`
|
||||
[Zenduty](#zenduty) | `webhook`
|
||||
|
||||
## Manage contact points for an external Alertmanager
|
||||
|
||||
Grafana alerting UI supports managing external Alertmanager configuration. Once you add an [Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page where you can select either `Grafana` or an external Alertmanager as your data source.
|
||||
|
||||
{{< figure max-width="40%" src="/static/img/docs/alerting/unified/contact-points-select-am-8-0.gif" caption="Select Alertmanager" >}}
|
||||
|
||||
### Edit Alertmanager global config
|
||||
|
||||
To edit global configuration options for an alertmanager, like SMTP server that is used by default for all email contact types:
|
||||
|
||||
|
||||
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
|
||||
1. In the dropdown at the top of the page, select an Alertmanager data source.
|
||||
1. Click **Edit global config** button at the bottom of the page.
|
||||
1. Fill out the form and click **Save global config**.
|
||||
|
||||
**Note** this is only for external Alertmanagers. Some global options for Grafana contact types, like email settings, can be configured via [Grafana configuration]({{< relref "../../administration/configuration.md" >}}).
|
||||
@@ -0,0 +1,67 @@
|
||||
+++
|
||||
title = "Grafana managed alert rules for numeric data"
|
||||
description = "Grafana managed alert rules for numeric data"
|
||||
keywords = ["grafana", "alerting", "guide", "rules", "create"]
|
||||
weight = 400
|
||||
+++
|
||||
|
||||
# Alerting on numeric data
|
||||
|
||||
Among certain data sources numeric data that is not time series can be directly alerted on, or passed into Server Side Expressions (SSE). This allows for more processing and resulting efficiency within the data source, and it can also simplify alert rules.
|
||||
When alerting on numeric data instead of time series data, there is no need to reduce each labeled time series into a single number. Instead labeled numbers are returned to Grafana instead.
|
||||
|
||||
## Tabular Data
|
||||
|
||||
This feature is supported with backend data sources that query tabular data:
|
||||
|
||||
- SQL data sources such as MySQL, Postgres, MSSQL, and Oracle.
|
||||
- The Azure Kusto based services: Azure Monitor (Logs), Azure Monitor (Azure Resource Graph), and Azure Data Explorer.
|
||||
|
||||
A query with Grafana managed alerts or SSE is considered numeric with these data sources, if:
|
||||
|
||||
- The "Format AS" option is set to "Table" in the data source query.
|
||||
- The table response returned to Grafana from the query includes only one numeric (e.g. int, double, float) column, and optionally additional string columns.
|
||||
|
||||
If there are string columns then those columns become labels. The name of column becomes the label name, and the value for each row becomes the value of the corresponding label. If multiple rows are returned, then each row should be uniquely identified their labels.
|
||||
|
||||
## Example
|
||||
|
||||
For a MySQL table called "DiskSpace":
|
||||
|
||||
| Time | Host | Disk | PercentFree
|
||||
| ----------- | --- | -----| --------
|
||||
| 2021-June-7 | web1 | /etc | 3
|
||||
| 2021-June-7 | web2 | /var | 4
|
||||
| 2021-June-7 | web3 | /var | 8
|
||||
| ... | ... | ... | ...
|
||||
|
||||
You can query the data filtering on time, but without returning the time series to Grafana. For example, an alert that would trigger per Host, Disk when there is less than 5% free space:
|
||||
|
||||
```sql
|
||||
SELECT Host, Disk, CASE WHEN PercentFree < 5.0 THEN PercentFree ELSE 0 END FROM (
|
||||
SELECT
|
||||
Host,
|
||||
Disk,
|
||||
Avg(PercentFree)
|
||||
FROM DiskSpace
|
||||
Group By
|
||||
Host,
|
||||
Disk
|
||||
Where __timeFilter(Time)
|
||||
```
|
||||
|
||||
This query returns the following Table response to Grafana:
|
||||
|
||||
| Host | Disk | PercentFree
|
||||
| --- | -----| --------
|
||||
| web1 | /etc | 3
|
||||
| web2 | /var | 4
|
||||
| web3 | /var | 0
|
||||
|
||||
When this query is used as the **condition** in an alert rule, then the non-zero will be alerting. As a result, three alert instances are produced:
|
||||
|
||||
| Labels | Status
|
||||
| ----------------------| ------
|
||||
| {Host=web1,disk=/etc} | Alerting
|
||||
| {Host=web2,disk=/var} | Alerting
|
||||
| {Host=web3,disk=/var} | Normal
|
||||
@@ -0,0 +1,102 @@
|
||||
+++
|
||||
title = "Message templating"
|
||||
description = "Message templating"
|
||||
aliases = ["/docs/grafana/latest/alerting/message-templating/"]
|
||||
keywords = ["grafana", "alerting", "guide", "contact point", "templating"]
|
||||
weight = 400
|
||||
+++
|
||||
|
||||
# Message templating
|
||||
|
||||
Notifications sent via [contact points]({{< relref "../contact-points.md" >}}) are built using templates. Grafana comes with default templates which you can customize. Grafana's notification templates are based on the [Go templating system](https://golang.org/pkg/text/template) where some fields are evaluated as text, while others are evaluated as HTML which can affect escaping. Since most of the contact point fields can be templated, you can create reusable templates and them in multiple contact points. See [template data reference]({{< relref "./template-data.md" >}}) to check what variables are available in the templates.
|
||||
|
||||
|
||||
## Using templating in contact point fields
|
||||
|
||||
This section shows an example of using templating to render a number of firing or resolved alerts in Slack message title, and listing alerts with status and name in the message body:
|
||||
|
||||
<img src="/static/img/docs/alerting/unified/contact-points-template-fields-8-0.png" width="600px">
|
||||
|
||||
## Reusable templates
|
||||
|
||||
You can create named templates and then reuse them in contact point fields or other templates.
|
||||
|
||||
### Create a template
|
||||
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
|
||||
1. Click **Add template**.
|
||||
1. Fill in **Name** and **Content** fields.
|
||||
1. Click **Save template** button at the bottom of the page.
|
||||
|
||||
**Note** The template name used to reference this template in templating is not the value of the **Name** field, but the parameter to `define` tag in the content. When creating a template you can omit `define` entirely and it will be added automatically with same value as **Name** field. It's recommended to use the same name for `define` and **Name** field to avoid confusion.
|
||||
|
||||
|
||||
<img src="/static/img/docs/alerting/unified/templates-create-8-0.png" width="600px">
|
||||
|
||||
### Edit a template
|
||||
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
|
||||
1. Find the template you want to edit in the templates table and click the **pen icon** on the right side.
|
||||
1. Make any changes and click **Save template** button at the bottom of the page.
|
||||
|
||||
### Delete a template
|
||||
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
|
||||
1. Find the template you want to edit in the templates table and click the **trash can icon** on the right side.
|
||||
1. A confirmation dialog will open. Click **Yes, delete**.
|
||||
|
||||
**Note** You are not prevented from deleting templates that are in use somewhere in contact points or other templates. Be careful!
|
||||
|
||||
### Use a template in a contact point field
|
||||
|
||||
To use a template:
|
||||
|
||||
Enter `{{ template "templatename" . }}` into a contact point field, where `templatename` is the `define` parameter of a template.
|
||||
|
||||
<img src="/static/img/docs/alerting/unified/contact-points-use-template-8-0.png" width="600px">
|
||||
|
||||
|
||||
### Template examples
|
||||
|
||||
Here is an example of a template to render a single alert:
|
||||
```
|
||||
{{ define "alert" }}
|
||||
[{{.Status}}] {{ .Labels.alertname }}
|
||||
|
||||
Labels:
|
||||
{{ range .Labels.SortedPairs }}
|
||||
{{ .Name }}: {{ .Value }}
|
||||
{{ end }}
|
||||
|
||||
{{ if gt (len .Annotations) 0 }}
|
||||
Annotations:
|
||||
{{ range .Annotations.SortedPairs }}
|
||||
{{ .Name }}: {{ .Value }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
|
||||
{{ if gt (len .SilenceURL ) 0 }}
|
||||
Silence alert: {{ .SilenceURL }}
|
||||
{{ end }}
|
||||
{{ if gt (len .DashboardURL ) 0 }}
|
||||
Go to dashboard: {{ .Dashboard URL }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
```
|
||||
|
||||
Template to render entire notification message:
|
||||
```
|
||||
{{ define "message" }}
|
||||
{{ if gt (len .Alerts.Firing) 0 }}
|
||||
{{ len .Alerts.Firing }} firing:
|
||||
{{ range .Alerts.Firing }} {{ template "alert" .}} {{ end }}
|
||||
{{ end }}
|
||||
{{ if gt (len .Alerts.Resolved) 0 }}
|
||||
{{ len .Alerts.Resolved }} resolved:
|
||||
{{ range .Alerts.Resolved }} {{ template "alert" .}} {{ end }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
```
|
||||
|
||||
## Manage templates for an external Alertmanager
|
||||
|
||||
Grafana alerting UI supports managing external Alertmanager configuration. Once you add an [Alertmanager data source]({{< relref "../../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page, allowing you to select either `Grafana` or an external Alertmanager data source.
|
||||
|
||||
{{< figure max-width="40%" src="/static/img/docs/alerting/unified/contact-points-select-am-8-0.gif" caption="Select Alertmanager" >}}
|
||||
@@ -0,0 +1,75 @@
|
||||
+++
|
||||
title = "Template data"
|
||||
keywords = ["grafana", "alerting", "guide", "contact point", "templating"]
|
||||
+++
|
||||
|
||||
|
||||
# Template data
|
||||
|
||||
Template data is passed on to [message templates]({{< relref "./_index.md" >}}) as well as sent as payload to webhook pushes.
|
||||
|
||||
Name | Type | Notes
|
||||
------------------|----------|-----------------------------------------------------------------
|
||||
Receiver | string | Name of the contact point that the notification is being sent to.
|
||||
Status | string | `firing` if at least one alert is firing, otherwise `resolved`.
|
||||
Alerts | Alert | List of alert objects that are included in this notification (see below).
|
||||
GroupLabels | KeyValue | Labels these alerts were grouped by.
|
||||
CommonLabels | KeyValue | Labels common to all the alerts included in this notification.
|
||||
CommonAnnotations | KeyValue | Annotations common to all the alerts included in this notification.
|
||||
ExternalURL | string | Back link to the Grafana that sent the notification. If using external Alertmanager, back link to this Alertmanager.
|
||||
|
||||
The `Alerts` type exposes functions for filtering alerts:
|
||||
|
||||
* `Alerts.Firing` returns a list of firing alerts.
|
||||
* `Alerts.Resolved` returns a list of resolved alerts.
|
||||
|
||||
## Alert
|
||||
|
||||
Name | Type | Notes
|
||||
-------------|-----------|---------------------------------------------------------------------
|
||||
Status | string | `firing` or `resolved`.
|
||||
Labels | KeyValue | A set of labels attached to the alert.
|
||||
Annotations | KeyValue | A set of annotations attached to the alert.
|
||||
StartsAt | time.Time | Time the alert started firing.
|
||||
EndsAt | time.Time | Only set if the end time of an alert is known. Otherwise set to a configurable timeout period from the time since the last alert was received.
|
||||
GeneratorURL | string | A back link to Grafana or external Alertmanager.
|
||||
SilenceURL | string | Link to grafana silence for with labels for this alert pre-filled. Only for Grafana managed alerts.
|
||||
DashboardURL | string | Link to grafana dashboard, if alert rule belongs to one. Only for Grafana managed alerts.
|
||||
PanelURL | string | Link to grafana dashboard panel, if alert rule belongs to one. Only for Grafana managed alerts.
|
||||
Fingerprint | string | Fingerprint that can be used to identify the alert.
|
||||
|
||||
## KeyValue
|
||||
|
||||
`KeyValue` is a set of key/value string pairs that represent labels and annotations.
|
||||
|
||||
Here is an example containing two annotations:
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": "alert summary",
|
||||
"description": "alert description"
|
||||
}
|
||||
```
|
||||
|
||||
In addition to direct access of data (labels and annotations) stored as KeyValue, there are also methods for sorting, removing and transforming.
|
||||
|
||||
Name | Arguments | Returns | Notes
|
||||
------------|-----------|-----------------------------------------|----------------
|
||||
SortedPairs | | Sorted list of key & value string pairs |
|
||||
Remove | []string | KeyValue | Returns a copy of the Key/Value map without the given keys.
|
||||
Names | | []string | List of label names
|
||||
Values | | []string | List of label values
|
||||
|
||||
## Functions
|
||||
|
||||
Some functions to transform values are also available, along with [default functions provided by Go templating](https://golang.org/pkg/text/template/#hdr-Functions).
|
||||
|
||||
Name | Arguments | Returns
|
||||
-------------|------------------------------|----------------------------------------------
|
||||
title | string | Capitalizes first character of each word.
|
||||
toUpper | string | Converts all characters to upper case.
|
||||
match | pattern, string | Match a string using RegExp.
|
||||
reReplaceAll | pattern, replacement, string | RegExp substitution, unanchored.
|
||||
join | string, []string | Concatenates the elements of the second argument to create a single string. First argument is the separator.
|
||||
safeHtml | string | Marks string as HTML, not requiring auto-escaping.
|
||||
stringSlice | ...string | Returns passed strings as slice of strings.
|
||||
@@ -0,0 +1,76 @@
|
||||
+++
|
||||
title = "Notification policies"
|
||||
description = "Notification policies"
|
||||
keywords = ["grafana", "alerting", "guide", "notification policies", "routes"]
|
||||
weight = 400
|
||||
+++
|
||||
|
||||
# Notification policies
|
||||
|
||||
Notification policies determine how alerts are routed to contact points. Policies have a tree structure, where each policy can have one or more child policies. Each policy except for the root policy can also match specific alert labels. Each alert enters policy tree at the root and then traverses each child policy. If `Continue matching subsequent sibling nodes` is not checked, it stops at the first matching node, otherwise, it continues matching it's siblings as well. If an alert does not match any children of a policy, the alert is handled based on the configuration settings of this policy and notified to the contact point configured on this policy. Alert that does not match any specific policy is handled by the root policy.
|
||||
|
||||
Grafana alerting UI allows you to configure Grafana notification policies as well as notification policies (routes) for an [external Alertmanager if one is configured]({{< relref "../../datasources/alertmanager.md" >}}).
|
||||
|
||||
|
||||
## Edit notification policies
|
||||
|
||||
To access notification policy editing page, In the Grafana side bar, hover your cursor over the **Alerting (bell)** icon and then click **Notification policies**.
|
||||
|
||||
### Edit root notification policy
|
||||
|
||||
1. Click **edit** button on the top right of the root policy box.
|
||||
1. Make changes and click **save** button.
|
||||
|
||||
### Add new specific policy
|
||||
|
||||
To add a top level specific policy, click **New policy** button in the **Specific routing** section, fill in the form and click **Save policy**.
|
||||
|
||||
To add a nested policy to an existing specific policy, expand the parent policy in specific routing table and click **Add nested policy**. fill in the form and click **Save policy**.
|
||||
|
||||
### Edit specific policy
|
||||
|
||||
To edit a specific policy, find it in the specific routing table and click **Edit** button. Make your changes and click **Save policy**.
|
||||
|
||||
|
||||
### Root policy fields
|
||||
|
||||
- **Default contact point -** The [contact point]({{< relref "./contact-points.md" >}}) to send notifications to that did not match any specific policy.
|
||||
- **Group by -** Labels to group alerts by. If multiple alerts are matched for this policy, they will be grouped based on these labels and a notification will be sent per group. Mandatory for root policy, optional for nested specific policies. If a specific policy does not specify own grouping, root policy grouping will be used.
|
||||
|
||||
Group timing options
|
||||
|
||||
- **Group wait -** - How long to wait to buffer alerts of the same group before sending a notification initially. Default is 30 seconds.
|
||||
- **Group interval -** - How long to wait before sending an notification when an alert has been added to a group for which there has already been a notification. Default is 5 minutes.
|
||||
- **Repeat interval -** - How long to wait before re-sending a notification after one has already been sent and no new alerts were added to the group. Default is 4 hours.
|
||||
|
||||
|
||||
### Specific policy fields
|
||||
|
||||
- **Contact point -** The [contact point]({{< relref "./contact-points.md" >}}) to send notification to if alert matched this specific policy but did not match any of it's nested policies, or there were no nested specific policies.
|
||||
- **Matching labels -** Rules for matching alert labels. See ["How label matching works"](#how-label-matching-works) below for details.
|
||||
- **Continue matching subsequent sibling nodes -** If not enabled and an alert matches this policy but not any of it's nested policies, matching will stop and a notification will be sent to the contact point defined on this policy. If enabled, notification will be sent but alert will continue matching subsequent siblings of this policy, thus sending more than one notification. Use this if for example you want to send notification to a catch-all contact point as well as to one of more specific contact points handled by subsequent policies.
|
||||
- **Override grouping** - Toggle if you want to override grouping for this policy. If toggled, you will be able to specify grouping same as for root policy described above. If not toggled, root policy grouping will be used.
|
||||
- **Override group timings** Toggle if you want to override group timings for this policy. If toggled, you will be able to specify group timings same as for root policy described above. If not toggled, root policy group timings will be used.
|
||||
|
||||
|
||||
### How label matching works
|
||||
|
||||
A policy will match an alert if alert's labels match all of the "Matching Labels" specified on the policy.
|
||||
|
||||
- The **Label** field is the name of the label to match. It must exactly match the label name.
|
||||
- The **Value** field matches against the corresponding value for the specified **Label** name. How it matches depends on the **Regex** and **Equal** checkboxes.
|
||||
- The **Regex** checkbox specifies if the inputted **Value** should be matched against labels as a regular expression. The regular expression is always anchored. If not selected it is an exact string match.
|
||||
- The **Equal** checkbox specifies if the match should include alert instances that match or do not match. If not checked, the silence includes alert instances _do not_ match.
|
||||
|
||||
|
||||
## Example setup
|
||||
|
||||
One usage example would be:
|
||||
* Create a "default" contact point for most alerts with a non invasive contact point type, like a slack message, and set it on root policy
|
||||
* Edit root policy grouping to group alerts by `cluster`, `namespace` and `alertname` so you get a notification per alert rule and specific k8s cluster & namespace.
|
||||
* Create specific route for alerts coming from development cluster with an appropriate contact point
|
||||
* Create a specific route for alerts with "critical" severity with a more invasive contact point type, like pager duty notification
|
||||
* Create specific routes for particular teams that handle their own onduty rotations
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
+++
|
||||
title = "Opt-in to Grafana 8 Alerts"
|
||||
description = "How to enable Grafana 8 Alerts"
|
||||
+++
|
||||
|
||||
# Enable Grafana 8 Alerts
|
||||
Setting the `ngalert` feature toggle enables the new Grafana 8 Alerting system.
|
||||
|
||||
>**Note:** It is recommended to backup Grafana's database before enabling this feature. If you are using PostgreSQL as the backend data source, then the minimum required version is 9.5.
|
||||
|
||||
At startup, when the feature toggle is enabled, Grafana dashboard alerting is disabled and existing dashboard alerts are migrated into a format that is compatible with the Grafana 8 Alerting system. You are able to view these migrated rules, alongside any new alerts you create after the migration, from the Alerting page of your grafana installation.
|
||||
Also notification channels are migrated to an Alertmanager configuration with the appropriate routes and receivers. Default notification channels are added as contact points to the default route. Notification channels not associated with any dahsboard alert go to the `autogen-unlinked-channel-recv` route.
|
||||
Since `Hipchat` and `Sensu` are discontinued, they are not migrated to the new alerting. If you have dashboard alerts associated with those types of channels and you want to migrate to the new alerting, make sure you assign another supported notification channel, so that you continue to receive notifications for those alerts.
|
||||
Finally, silences (expiring after one year) are created for all paused dashboard alerts.
|
||||
|
||||
During beta, the migration of existing dashboard rules may change.
|
||||
|
||||
## Disabling Grafana 8 Alerting after migration
|
||||
To disable Grafana 8 Alerting, remove or disable the `ngalert` feature toggle. Dashboard alerts will be re-enabled and any alerts created during or after the migration are deleted.
|
||||
|
||||
>**Note:** Any alerting rules created in the Grafana 8 Alerting system will be lost when migrating back to dashboard alerts
|
||||
@@ -0,0 +1,41 @@
|
||||
+++
|
||||
title = "Silence alert notifications"
|
||||
description = "Silence alert notifications"
|
||||
keywords = ["grafana", "alerting", "silence", "mute"]
|
||||
weight = 400
|
||||
+++
|
||||
|
||||
# Silence alert notifications
|
||||
|
||||
Grafana allows to you to prevent notifications from one or more alert rules by creating a silence. This silence lasts for a specified window of time.
|
||||
|
||||
Silences do not prevent alert rules from being evaluated. They also do not stop alert instances being shown in the user interface. Silences only prevent notifications from being created.
|
||||
|
||||
## Add a silence
|
||||
|
||||
To add a silence:
|
||||
|
||||
1. In the Grafana menu, hover your cursor over the **Alerting** (bell) icon and then select **Silences** (crossed out bell icon).
|
||||
1. Click the **New Silence** button.
|
||||
1. Select the start and end date in **Silence start and end** to indicate when the silence should go into effect and expire.
|
||||
1. Optionally, update the **Duration** to alter the time for the end of silence in the previous step to correspond to the start plus the duration.
|
||||
1. Enter one or more *Matching Labels* by filling out the **Name** and **Value** fields. Matchers determine which rules the silence will apply to.
|
||||
1. Enter a **Comment**.
|
||||
1. Enter the name of the owner in **Creator**.
|
||||
1. Click **Create**.
|
||||
|
||||
## How label matching works
|
||||
|
||||
Alert instances that have labels that match all of the "Matching Labels" specified in the silence will have their notifications suppressed.
|
||||
|
||||
- The **Label** field is the name of the label to match. It must exactly match the label name.
|
||||
- The **Value** field matches against the corresponding value for the specified **Label** name. How it matches depends on the **Regex** and **Equal** checkboxes.
|
||||
- The **Regex** checkbox specifies if the inputted **Value** should be matched against labels as a regular expression. The regular expression is always anchored. If not selected it is an exact string match.
|
||||
- The **Equal** checkbox specifies if the match should include alert instances that match or do not match. If not checked, the silence includes alert instances _do not_ match.
|
||||
|
||||
## Viewing and editing silences
|
||||
|
||||
1. In the Grafana menu hover your cursor over the **Alerting** (bell) icon, then select **Silences** (crossed out bell icon).
|
||||
1. To end the silence, click the **Unsilence** option next to the listed silence. Silences that have ended are still listed and are automatically removed after 5 days. There is no method for manual removal.
|
||||
1. To edit a silence, click the pencil icon next to the listed silence. Edit the silence using instructions on how to create a silence.
|
||||
1. Click **Submit** to save your changes.
|
||||
Reference in New Issue
Block a user