public/grafana
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

PackageJson: Prettify markdown/mdx on commit with lint-staged (#37616)

Browse Source 
* Format md,mdx files with prettier on lint-staged

* Manually run prettier on docs/sources
This commit is contained in:
Connor Lindsey
2021-08-06 07:52:36 -06:00
committed by GitHub
parent e9c032f10f
commit b78a67cec7
301 changed files with 3216 additions and 2980 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sources/alerting/_index.md
+1 -1
View File
@@ -26,4 +26,4 @@ As part of the new alert changes, we have introduced a new data source, Alertman
> **Note:** Out of the box, Grafana still supports old Grafana alerts. They are legacy alerts at this time, and will be deprecated in a future release. For more information, refer to [Legacy Grafana alerts]({{< relref "./old-alerting/_index.md" >}}).
To learn more about the differences between new alerts and the legacy alerts, refer to [What's New with Grafana 8 Alerts]({{< relref "../alerting/difference-old-new.md" >}}).
To learn more about the differences between new alerts and the legacy alerts, refer to [What's New with Grafana 8 Alerts]({{< relref "../alerting/difference-old-new.md" >}}).
docs/sources/alerting/difference-old-new.md
+8 -3
View File
@@ -6,16 +6,21 @@ weight = 112
+++
# What's New with Grafana 8 Alerts
The Alerts released with Grafana 8.0 are an opt-in feature that centralizes alerting information for Grafana managed alerts and alerts from Prometheus-compatible datasources in one UI and API. You are able to create and edit alerting rules for Grafana managed alerts, Cortex alerts, and Loki alerts as well as see alerting information from prometheus-compatible datasources in a single, searchable view.
## Multi-dimensional alerting
Create alerts that will give you system-wide visibility with a single alerting rule. With Grafana 8 alerts, you are able to generate multiple alert instances from a single rule eg. creating a rule to monitor disk usage for multiple mount points on a single host. The evaluation engine is able to return multiple time series from a single query. Each time series is identified by its label set.
Create alerts that will give you system-wide visibility with a single alerting rule. With Grafana 8 alerts, you are able to generate multiple alert instances from a single rule eg. creating a rule to monitor disk usage for multiple mount points on a single host. The evaluation engine is able to return multiple time series from a single query. Each time series is identified by its label set.
## Create alerts outside of Dashboards
Grafana legacy alerts were tied to a dashboard. Grafana 8 Alerts allow you to create queries and expressions that can combine data from multiple sources, in unique ways. You are still able to link dashboards and panels to alerting rules, allowing you to quickly troubleshoot the system under observation, by linking a dashboard and/or panel ID to the alerting rule.
Grafana legacy alerts were tied to a dashboard. Grafana 8 Alerts allow you to create queries and expressions that can combine data from multiple sources, in unique ways. You are still able to link dashboards and panels to alerting rules, allowing you to quickly troubleshoot the system under observation, by linking a dashboard and/or panel ID to the alerting rule.
## Create Loki and Cortex alerting rules
With Grafana 8 Alerts you are able to manage your Loki and Cortex alerting rules using the same UI and API as your Grafana managed alerts.
With Grafana 8 Alerts you are able to manage your Loki and Cortex alerting rules using the same UI and API as your Grafana managed alerts.
## View and search for alerts from Prometheus
You can now display all of your alerting information in one, searchable UI. Alerts for Prometheus compatible datasources are listed below Grafana managed alerts. Search for labels across multiple datasources to quickly find all of the relevant alerts.
docs/sources/alerting/old-alerting/_index.md
+12 -11
View File
@@ -17,7 +17,7 @@ Currently only the graph panel visualization supports alerts.
Legacy alerts have two main components:
- Alert rule - When the alert is triggered. Alert rules are defined by one or more conditions that are regularly evaluated by Grafana.
- Notification channel - How the alert is delivered. When the conditions of an alert rule are met, the Grafana notifies the channels configured for that alert.
- Notification channel - How the alert is delivered. When the conditions of an alert rule are met, the Grafana notifies the channels configured for that alert.
## Alert tasks
@@ -37,20 +37,21 @@ Currently alerting supports a limited form of high availability. Since v4.2.0 of
Grafana managed alerts are evaluated by the Grafana backend. Rule evaluations are scheduled, according to the alert rule configuration, and queries are evaluated by an engine that is part of core Grafana.
Alert rules can only query backend data sources with alerting enabled:
- builtin or developed and maintained by grafana: `Graphite`, `Prometheus`, `Loki`, `InfluxDB`, `Elasticsearch`,
`Google Cloud Monitoring`, `Cloudwatch`, `Azure Monitor`, `MySQL`, `PostgreSQL`, `MSSQL`, `OpenTSDB`, `Oracle`, and `Azure Data Explorer`
`Google Cloud Monitoring`, `Cloudwatch`, `Azure Monitor`, `MySQL`, `PostgreSQL`, `MSSQL`, `OpenTSDB`, `Oracle`, and `Azure Data Explorer`
- any community backend data sources with alerting enabled (`backend` and `alerting` properties are set in the [plugin.json]({{< relref "../../developers/plugins/metadata.md" >}}))
## Metrics from the alert engine
The alert engine publishes some internal metrics about itself. You can read more about how Grafana publishes [internal metrics]({{< relref "../../administration/view-server/internal-metrics.md" >}}).
Metric Name | Type | Description
---------- | ----------- | ----------
`alerting.alerts` | gauge | How many alerts by state
`alerting.request_duration_seconds` | histogram | Histogram of requests to the Alerting API
`alerting.active_configurations` | gauge | The number of active, non default alertmanager configurations for grafana managed alerts
`alerting.rule_evaluations_total` | counter | The total number of rule evaluations
`alerting.rule_evaluation_failures_total` | counter | The total number of rule evaluation failures
`alerting.rule_evaluation_duration_seconds` | summary | The duration for a rule to execute
`alerting.rule_group_rules` | gauge | The number of rules
| Metric Name | Type | Description |
| ------------------------------------------- | --------- | ---------------------------------------------------------------------------------------- |
| `alerting.alerts` | gauge | How many alerts by state |
| `alerting.request_duration_seconds` | histogram | Histogram of requests to the Alerting API |
| `alerting.active_configurations` | gauge | The number of active, non default alertmanager configurations for grafana managed alerts |
| `alerting.rule_evaluations_total` | counter | The total number of rule evaluations |
| `alerting.rule_evaluation_failures_total` | counter | The total number of rule evaluation failures |
| `alerting.rule_evaluation_duration_seconds` | summary | The duration for a rule to execute |
| `alerting.rule_group_rules` | gauge | The number of rules |
docs/sources/alerting/old-alerting/create-alerts.md
+4 -5
View File
@@ -96,16 +96,16 @@ Below are conditions you can configure how the rule evaluation engine should han
| --------------- | ------------------------------------------------------------------------------------------ |
| No Data | Set alert rule state to `NoData` |
| Alerting | Set alert rule state to `Alerting` |
| Keep Last State | Keep the current alert rule state, whatever it is. |
| Keep Last State | Keep the current alert rule state, whatever it is. |
| Ok | Not sure why you would want to send yourself an alert when things are okay, but you could. |
### Execution errors or timeouts
Tell Grafana how to handle execution or timeout errors.
| Error or timeout option | Description |
| ----------------------- | --------------------------------------------------- |
| Alerting | Set alert rule state to `Alerting` |
| Error or timeout option | Description |
| ----------------------- | -------------------------------------------------- |
| Alerting | Set alert rule state to `Alerting` |
| Keep Last State | Keep the current alert rule state, whatever it is. |
If you have an unreliable time series store from which queries sometime timeout or fail randomly you can set this option to `Keep Last State` in order to basically ignore them.
@@ -124,4 +124,3 @@ The actual notifications are configured and shared between multiple alerts. Read
## Alert state history and annotations
Alert state changes are recorded in the internal annotation table in Grafana's database. The state changes are visualized as annotations in the alert rule's graph panel. You can also go into the `State history` submenu in the alert tab to view and clear state history.
docs/sources/alerting/old-alerting/notifications.md
+96 -96
View File
@@ -37,41 +37,41 @@ This is done from the Notification channels page.
These examples show how often and when reminders are sent for a triggered alert.
Alert rule evaluation interval | Send reminders every | Reminder sent every (after last alert notification)
---------- | ----------- | -----------
`30s` | `15s` | ~30 seconds
`1m` | `5m` | ~5 minutes
`5m` | `15m` | ~15 minutes
`6m` | `20m` | ~24 minutes
`1h` | `15m` | ~1 hour
`1h` | `2h` | ~2 hours
| Alert rule evaluation interval | Send reminders every | Reminder sent every (after last alert notification) |
| ------------------------------ | -------------------- | --------------------------------------------------- |
| `30s` | `15s` | ~30 seconds |
| `1m` | `5m` | ~5 minutes |
| `5m` | `15m` | ~15 minutes |
| `6m` | `20m` | ~24 minutes |
| `1h` | `15m` | ~1 hour |
| `1h` | `2h` | ~2 hours |
<div class="clearfix"></div>
## List of supported notifiers
Name | Type | Supports images | Support alert rule tags
-----|------|---------------- | -----------------------
[DingDing](#dingdingdingtalk) | `dingding` | yes, external only | no
[Discord](#discord) | `discord` | yes | no
[Email](#email) | `email` | yes | no
[Google Hangouts Chat](#google-hangouts-chat) | `googlechat` | yes, external only | no
Hipchat | `hipchat` | yes, external only | no
[Kafka](#kafka) | `kafka` | yes, external only | no
Line | `line` | yes, external only | no
Microsoft Teams | `teams` | yes, external only | no
[Opsgenie](#opsgenie) | `opsgenie` | yes, external only | yes
[Pagerduty](#pagerduty) | `pagerduty` | yes, external only | yes
Prometheus Alertmanager | `prometheus-alertmanager` | yes, external only | yes
[Pushover](#pushover) | `pushover` | yes | no
Sensu | `sensu` | yes, external only | no
[Sensu Go](#sensu-go) | `sensugo` | yes, external only | no
[Slack](#slack) | `slack` | yes | no
Telegram | `telegram` | yes | no
Threema | `threema` | yes, external only | no
VictorOps | `victorops` | yes, external only | yes
[Webhook](#webhook) | `webhook` | yes, external only | yes
[Zenduty](#zenduty) | `webhook` | yes, external only | yes
| Name | Type | Supports images | Support alert rule tags |
| --------------------------------------------- | ------------------------- | ------------------ | ----------------------- |
| [DingDing](#dingdingdingtalk) | `dingding` | yes, external only | no |
| [Discord](#discord) | `discord` | yes | no |
| [Email](#email) | `email` | yes | no |
| [Google Hangouts Chat](#google-hangouts-chat) | `googlechat` | yes, external only | no |
| Hipchat | `hipchat` | yes, external only | no |
| [Kafka](#kafka) | `kafka` | yes, external only | no |
| Line | `line` | yes, external only | no |
| Microsoft Teams | `teams` | yes, external only | no |
| [Opsgenie](#opsgenie) | `opsgenie` | yes, external only | yes |
| [Pagerduty](#pagerduty) | `pagerduty` | yes, external only | yes |
| Prometheus Alertmanager | `prometheus-alertmanager` | yes, external only | yes |
| [Pushover](#pushover) | `pushover` | yes | no |
| Sensu | `sensu` | yes, external only | no |
| [Sensu Go](#sensu-go) | `sensugo` | yes, external only | no |
| [Slack](#slack) | `slack` | yes | no |
| Telegram | `telegram` | yes | no |
| Threema | `threema` | yes, external only | no |
| VictorOps | `victorops` | yes, external only | yes |
| [Webhook](#webhook) | `webhook` | yes, external only | yes |
| [Zenduty](#zenduty) | `webhook` | yes, external only | yes |
### Email
@@ -83,10 +83,10 @@ able to access the image.
> **Note:** Template variables are not supported in email alerts.
Setting | Description
---------- | -----------
Single email | Send a single email to all recipients. Disabled per default.
Addresses | Email addresses to recipients. You can enter multiple email addresses using a ";" separator.
| Setting | Description |
| ------------ | -------------------------------------------------------------------------------------------- |
| Single email | Send a single email to all recipients. Disabled per default. |
| Addresses | Email addresses to recipients. You can enter multiple email addresses using a ";" separator. |
### Slack
@@ -98,17 +98,17 @@ firing alerts in the Slack messages you have to configure either the [external i
in Grafana or a bot integration via Slack Apps. [Follow Slack's guide to set up a bot integration](https://api.slack.com/bot-users) and use the token
provided, which starts with "xoxb".
Setting | Description
---------- | -----------
Url | Slack incoming webhook URL, or eventually the [chat.postMessage](https://api.slack.com/methods/chat.postMessage) Slack API endpoint.
Username | Set the username for the bot's message.
Recipient | Allows you to override the Slack recipient. You must either provide a channel Slack ID, a user Slack ID, a username reference (@&lt;user&gt;, all lowercase, no whitespace), or a channel reference (#&lt;channel&gt;, all lowercase, no whitespace). If you use the `chat.postMessage` Slack API endpoint, this is required.
Icon emoji | Provide an emoji to use as the icon for the bot's message. Ex :smile:
Icon URL | Provide a URL to an image to use as the icon for the bot's message.
Mention Users | Optionally mention one or more users in the Slack notification sent by Grafana. You have to refer to users, comma-separated, via their corresponding Slack IDs (which you can find by clicking the overflow button on each user's Slack profile).
Mention Groups | Optionally mention one or more groups in the Slack notification sent by Grafana. You have to refer to groups, comma-separated, via their corresponding Slack IDs (which you can get from each group's Slack profile URL).
Mention Channel | Optionally mention either all channel members or just active ones.
Token | If provided, Grafana will upload the generated image via Slack's file.upload API method, not the external image destination. If you use the `chat.postMessage` Slack API endpoint, this is required.
| Setting | Description |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Url | Slack incoming webhook URL, or eventually the [chat.postMessage](https://api.slack.com/methods/chat.postMessage) Slack API endpoint. |
| Username | Set the username for the bot's message. |
| Recipient | Allows you to override the Slack recipient. You must either provide a channel Slack ID, a user Slack ID, a username reference (@&lt;user&gt;, all lowercase, no whitespace), or a channel reference (#&lt;channel&gt;, all lowercase, no whitespace). If you use the `chat.postMessage` Slack API endpoint, this is required. |
| Icon emoji | Provide an emoji to use as the icon for the bot's message. Ex :smile: |
| Icon URL | Provide a URL to an image to use as the icon for the bot's message. |
| Mention Users | Optionally mention one or more users in the Slack notification sent by Grafana. You have to refer to users, comma-separated, via their corresponding Slack IDs (which you can find by clicking the overflow button on each user's Slack profile). |
| Mention Groups | Optionally mention one or more groups in the Slack notification sent by Grafana. You have to refer to groups, comma-separated, via their corresponding Slack IDs (which you can get from each group's Slack profile URL). |
| Mention Channel | Optionally mention either all channel members or just active ones. |
| Token | If provided, Grafana will upload the generated image via Slack's file.upload API method, not the external image destination. If you use the `chat.postMessage` Slack API endpoint, this is required. |
If you are using the token for a slack bot, then you have to invite the bot to the channel you want to send notifications and add the channel to the recipient field.
@@ -116,12 +116,12 @@ If you are using the token for a slack bot, then you have to invite the bot to t
To setup Opsgenie you will need an API Key and the Alert API Url. These can be obtained by configuring a new [Grafana Integration](https://docs.opsgenie.com/docs/grafana-integration).
Setting | Description
--------|------------
Alert API URL | The API URL for your Opsgenie instance. This will normally be either `https://api.opsgenie.com` or, for EU customers, `https://api.eu.opsgenie.com`.
API Key | The API Key as provided by Opsgenie for your configured Grafana integration.
Override priority | Configures the alert priority using the `og_priority` tag. The `og_priority` tag must have one of the following values: `P1`, `P2`, `P3`, `P4`, or `P5`. Default is `False`.
Send notification tags as | Specify how you would like [Notification Tags]({{< relref "create-alerts.md/#notifications" >}}) delivered to Opsgenie. They can be delivered as `Tags`, `Extra Properties` or both. Default is Tags. See note below for more information.
| Setting | Description |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Alert API URL | The API URL for your Opsgenie instance. This will normally be either `https://api.opsgenie.com` or, for EU customers, `https://api.eu.opsgenie.com`. |
| API Key | The API Key as provided by Opsgenie for your configured Grafana integration. |
| Override priority | Configures the alert priority using the `og_priority` tag. The `og_priority` tag must have one of the following values: `P1`, `P2`, `P3`, `P4`, or `P5`. Default is `False`. |
| Send notification tags as | Specify how you would like [Notification Tags]({{< relref "create-alerts.md/#notifications" >}}) delivered to Opsgenie. They can be delivered as `Tags`, `Extra Properties` or both. Default is Tags. See note below for more information. |
> **Note:** When notification tags are sent as `Tags` they are concatenated into a string with a `key:value` format. If you prefer to receive the notifications tags as key/values under Extra Properties in Opsgenie then change the `Send notification tags as` to either `Extra Properties` or `Tags & Extra Properties`.
@@ -129,19 +129,19 @@ Send notification tags as | Specify how you would like [Notification Tags]({{< r
To set up PagerDuty, all you have to do is to provide an integration key.
Setting | Description
---------- | -----------
Integration Key | Integration key for PagerDuty.
Severity | Level for dynamic notifications, default is `critical` (1)
Auto resolve incidents | Resolve incidents in PagerDuty once the alert goes back to ok
Message in details | Removes the Alert message from the PD summary field and puts it into custom details instead (2)
| Setting | Description |
| ---------------------- | ----------------------------------------------------------------------------------------------- |
| Integration Key | Integration key for PagerDuty. |
| Severity | Level for dynamic notifications, default is `critical` (1) |
| Auto resolve incidents | Resolve incidents in PagerDuty once the alert goes back to ok |
| Message in details | Removes the Alert message from the PD summary field and puts it into custom details instead (2) |
>**Note:** The tags `Severity`, `Class`, `Group`, `dedup_key`, and `Component` have special meaning in the [Pagerduty Common Event Format - PD-CEF](https://support.pagerduty.com/docs/pd-cef). If an alert panel defines these tag keys, then they are transposed to the root of the event sent to Pagerduty. This means they will be available within the Pagerduty UI and Filtering tools. A Severity tag set on an alert overrides the global Severity set on the notification channel if it's a valid level.
> **Note:** The tags `Severity`, `Class`, `Group`, `dedup_key`, and `Component` have special meaning in the [Pagerduty Common Event Format - PD-CEF](https://support.pagerduty.com/docs/pd-cef). If an alert panel defines these tag keys, then they are transposed to the root of the event sent to Pagerduty. This means they will be available within the Pagerduty UI and Filtering tools. A Severity tag set on an alert overrides the global Severity set on the notification channel if it's a valid level.
>Using Message In Details will change the structure of the `custom_details` field in the PagerDuty Event.
This might break custom event rules in your PagerDuty rules if you rely on the fields in `payload.custom_details`.
Move any existing rules using `custom_details.myMetric` to `custom_details.queries.myMetric`.
This behavior will become the default in a future version of Grafana.
> Using Message In Details will change the structure of the `custom_details` field in the PagerDuty Event.
> This might break custom event rules in your PagerDuty rules if you rely on the fields in `payload.custom_details`.
> Move any existing rules using `custom_details.myMetric` to `custom_details.queries.myMetric`.
> This behavior will become the default in a future version of Grafana.
> **Note:** The `dedup_key` tag overrides the Grafana-generated `dedup_key` with a custom key.
@@ -152,22 +152,22 @@ This behavior will become the default in a future version of Grafana.
To configure VictorOps, provide the URL from the Grafana Integration and substitute `$routing_key` with a valid key.
> **Note:** The tag `Severity` has special meaning in the [VictorOps Incident Fields](https://help.victorops.com/knowledge-base/incident-fields-glossary/). If an alert panel defines this key, then it replaces the `message_type` in the root of the event sent to VictorOps.
### Pushover
To set up Pushover, you must provide a user key and an API token. Refer to [What is Pushover and how do I use it](https://support.pushover.net/i7-what-is-pushover-and-how-do-i-use-it) for instructions on how to generate them.
Setting | Description
---------- | -----------
API Token | Application token
User key(s) | A comma-separated list of user keys
Device(s) | A comma-separated list of devices
Priority | The priority alerting nottifications are sent
OK priority | The priority OK notifications are sent; if not set, then OK notifications are sent with the priority set for alerting notifications
Retry | How often (in seconds) the Pushover servers send the same notification to the user. (minimum 30 seconds)
Expire | How many seconds your notification will continue to be retried for (maximum 86400 seconds)
Alerting sound | The sound for alerting notifications
OK sound | The sound for OK notifications
| Setting | Description |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| API Token | Application token |
| User key(s) | A comma-separated list of user keys |
| Device(s) | A comma-separated list of devices |
| Priority | The priority alerting nottifications are sent |
| OK priority | The priority OK notifications are sent; if not set, then OK notifications are sent with the priority set for alerting notifications |
| Retry | How often (in seconds) the Pushover servers send the same notification to the user. (minimum 30 seconds) |
| Expire | How many seconds your notification will continue to be retried for (maximum 86400 seconds) |
| Alerting sound | The sound for alerting notifications |
| OK sound | The sound for OK notifications |
### Webhook
@@ -178,26 +178,26 @@ Example json body:
```json
{
"dashboardId":1,
"evalMatches":[
"dashboardId": 1,
"evalMatches": [
{
"value":1,
"metric":"Count",
"tags":{}
"value": 1,
"metric": "Count",
"tags": {}
}
],
"imageUrl":"https://grafana.com/assets/img/blog/mixed_styles.png",
"message":"Notification Message",
"orgId":1,
"panelId":2,
"ruleId":1,
"ruleName":"Panel Title alert",
"ruleUrl":"http://localhost:3000/d/hZ7BuVbWz/test-dashboard?fullscreen\u0026edit\u0026tab=alert\u0026panelId=2\u0026orgId=1",
"state":"alerting",
"tags":{
"tag name":"tag value"
"imageUrl": "https://grafana.com/assets/img/blog/mixed_styles.png",
"message": "Notification Message",
"orgId": 1,
"panelId": 2,
"ruleId": 1,
"ruleName": "Panel Title alert",
"ruleUrl": "http://localhost:3000/d/hZ7BuVbWz/test-dashboard?fullscreen\u0026edit\u0026tab=alert\u0026panelId=2\u0026orgId=1",
"state": "alerting",
"tags": {
"tag name": "tag value"
},
"title":"[Alerting] Panel Title alert"
"title": "[Alerting] Panel Title alert"
}
```
@@ -213,7 +213,7 @@ In DingTalk PC Client:
2. Click "Robot Manage" item in the pop menu, there will be a new panel call "Robot Manage".
3. In the "Robot Manage" panel, select "customized: customized robot with Webhook".
3. In the "Robot Manage" panel, select "customized: customized robot with Webhook".
4. In the next new panel named "robot detail", click "Add" button.
@@ -226,11 +226,11 @@ In DingTalk PC Client:
To set up Discord, you must create a Discord channel webhook. For instructions on how to create the channel, refer to
[Intro to Webhooks](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks).
Setting | Description
---------- | -----------
Webhook URL | Discord webhook URL.
Message Content | Mention a group using @ or a user using <@ID> when notifying in a channel.
Avatar URL | Optionally, provide a URL to an image to use as the avatar for the bot's message.
| Setting | Description |
| --------------- | --------------------------------------------------------------------------------- |
| Webhook URL | Discord webhook URL. |
| Message Content | Mention a group using @ or a user using <@ID> when notifying in a channel. |
| Avatar URL | Optionally, provide a URL to an image to use as the avatar for the bot's message. |
Alternately, use the [Slack](#slack) notifier by appending `/slack` to a Discord webhook URL.
docs/sources/alerting/old-alerting/pause-an-alert-rule.md
+1 -1
View File
@@ -12,4 +12,4 @@ Pausing the evaluation of an alert rule can sometimes be useful. For example, du
1. In the Grafana side bar, hover your cursor over the Alerting (bell) icon and then click **Alert Rules**. All configured alert rules are listed, along with their current state.
1. Find your alert in the list, and click the **Pause** icon on the right. The **Pause** icon turns into a **Play** icon.
1. Click the **Play** icon to resume evaluation of your alert.
1. Click the **Play** icon to resume evaluation of your alert.
docs/sources/alerting/old-alerting/view-alerts.md
+1 -1
View File
@@ -17,4 +17,4 @@ You can do several things while viewing alerts.
- **Filter alerts by name -** Type an alert name in the **Search alerts** field.
- **Filter alerts by state -** In **States**, select which alert states you want to see. All others will be hidden.
- **Pause or resume an alert -** Click the **Pause** or **Play** icon next to the alert to pause or resume evaluation. See [Pause an alert rule]({{< relref "pause-an-alert-rule.md" >}}) for more information.
- **Access alert rule settings -** Click the alert name or the **Edit alert rule** (gear) icon. Grafana opens the Alert tab of the panel where the alert rule is defined. This is helpful when an alert is firing but you don't know which panel it is defined in.
- **Access alert rule settings -** Click the alert name or the **Edit alert rule** (gear) icon. Grafana opens the Alert tab of the panel where the alert rule is defined. This is helpful when an alert is firing but you don't know which panel it is defined in.
docs/sources/alerting/unified-alerting/_index.md
+12 -12
View File
@@ -5,9 +5,10 @@ weight = 113
+++
# Overview of Grafana 8 alerts
Alerts allow you to know about problems in your systems moments after they occur. Robust and actionable alerts help you identify and resolve issues quickly, minimizing disruption to your services.
>**Note:** This information is for the new, Grafana 8 Alerts. This is an [opt-in]({{< relref"./opt-in.md" >}}) feature released in Grafana 8.0. Grafana still supports [legacy dashboard alerts]({{< relref "../old-alerting/_index.md" >}}) out of the box
> **Note:** This information is for the new, Grafana 8 Alerts. This is an [opt-in]({{< relref"./opt-in.md" >}}) feature released in Grafana 8.0. Grafana still supports [legacy dashboard alerts]({{< relref "../old-alerting/_index.md" >}}) out of the box
Alerts have four main components:
@@ -20,7 +21,6 @@ Alerts have four main components:
You can perform the following tasks for alerts:
- [Create a Grafana managed alert rule]({{< relref "alerting-rules/create-grafana-managed-rule.md" >}})
- [Create a Cortex or Loki managed alert rule]({{< relref "alerting-rules/create-cortex-loki-managed-rule.md" >}})
- [View existing alert rules and their current state]({{< relref "alerting-rules/rule-list.md" >}})
@@ -38,6 +38,7 @@ The current alerting system doesn't support high availability. Alert notificatio
Grafana managed alerts are evaluated by the Grafana backend. Rule evaluations are scheduled, according to the alert rule configuration, and queries are evaluated by an engine that is part of core Grafana.
Alerting rules can only query backend data sources with alerting enabled:
- builtin or developed and maintained by grafana: `Graphite`, `Prometheus`, `Loki`, `InfluxDB`, `Elasticsearch`,
`Google Cloud Monitoring`, `Cloudwatch`, `Azure Monitor`, `MySQL`, `PostgreSQL`, `MSSQL`, `OpenTSDB`, `Oracle`, and `Azure Data Explorer`
- any community backend data sources with alerting enabled (`backend` and `alerting` properties are set in the [plugin.json]({{< relref "../../developers/plugins/metadata.md" >}}))
@@ -46,15 +47,14 @@ Alerting rules can only query backend data sources with alerting enabled:
The alerting engine publishes some internal metrics about itself. You can read more about how Grafana publishes [internal metrics]({{< relref "../../administration/view-server/internal-metrics.md" >}}).
Metric Name | Type | Description
---------- | ----------- | ----------
`alerting.alerts` | gauge | How many alerts by state
`alerting.request_duration_seconds` | histogram | Histogram of requests to the Alerting API
`alerting.active_configurations` | gauge | The number of active, non default Alertmanager configurations for grafana managed alerts
`alerting.rule_evaluations_total` | counter | The total number of rule evaluations
`alerting.rule_evaluation_failures_total` | counter | The total number of rule evaluation failures
`alerting.rule_evaluation_duration_seconds` | summary | The duration for a rule to execute
`alerting.rule_group_rules` | gauge | The number of rules
| Metric Name | Type | Description |
| ------------------------------------------- | --------- | ---------------------------------------------------------------------------------------- |
| `alerting.alerts` | gauge | How many alerts by state |
| `alerting.request_duration_seconds` | histogram | Histogram of requests to the Alerting API |
| `alerting.active_configurations` | gauge | The number of active, non default Alertmanager configurations for grafana managed alerts |
| `alerting.rule_evaluations_total` | counter | The total number of rule evaluations |
| `alerting.rule_evaluation_failures_total` | counter | The total number of rule evaluation failures |
| `alerting.rule_evaluation_duration_seconds` | summary | The duration for a rule to execute |
| `alerting.rule_group_rules` | gauge | The number of rules |
- [View alert rules and their current state]({{< relref "alerting-rules/rule-list.md" >}})
docs/sources/alerting/unified-alerting/alerting-rules/_index.md
+2 -1
View File
@@ -5,7 +5,8 @@ weight = 130
+++
# Create and manage alerting Rules
One or more queries and/or expressions, a condition, the frequency of evaluation, and the (optional) duration that a condition must be met before creating an alert. Alerting rules are how you express the criteria for creating an alert. Queries and expressions select and can operate on the data you wish to alert on. A condition sets the threshold that an alert must meet or exceed to create an alert. The interval specifies how frequently the rule should be evaluated. The duration, when configured, sets a period that a condition must be met or exceeded before an alert is created. Alerting rules also can contain settings for what to do when your query does not return any data, or there is an error attempting to execute the query.
One or more queries and/or expressions, a condition, the frequency of evaluation, and the (optional) duration that a condition must be met before creating an alert. Alerting rules are how you express the criteria for creating an alert. Queries and expressions select and can operate on the data you wish to alert on. A condition sets the threshold that an alert must meet or exceed to create an alert. The interval specifies how frequently the rule should be evaluated. The duration, when configured, sets a period that a condition must be met or exceeded before an alert is created. Alerting rules also can contain settings for what to do when your query does not return any data, or there is an error attempting to execute the query.
- [Create Cortex or Loki managed alert rule]({{< relref "./create-cortex-loki-managed-rule.md" >}})
- [Create Grafana managed alert rule]({{< relref "./create-grafana-managed-rule.md" >}})
docs/sources/alerting/unified-alerting/alerting-rules/create-cortex-loki-managed-rule.md
+10 -11
View File
@@ -7,10 +7,9 @@ weight = 400
# Create a Cortex or Loki managed alerting rule
Grafana allows you manage alerting rules for an external Cortex or Loki instance.
Grafana allows you manage alerting rules for an external Cortex or Loki instance.
In order for both Cortex and Loki data sources to work with Grafana 8.0 alerting, enable the ruler API by configuring their respective services. The`local` rule storage type, default for Loki, only supports viewing of rules. If you want to edit rules, then configure one of the other rule storage types. When configuring a Grafana Prometheus data source to point to Cortex, use the legacy `/api/prom` prefix, not `/prometheus`. Only single-binary mode is currently supported, and it is not possible to provide a separate URL for the ruler API.
In order for both Cortex and Loki data sources to work with Grafana 8.0 alerting, enable the ruler API by configuring their respective services. The`local` rule storage type, default for Loki, only supports viewing of rules. If you want to edit rules, then configure one of the other rule storage types. When configuring a Grafana Prometheus data source to point to Cortex, use the legacy `/api/prom` prefix, not `/prometheus`. Only single-binary mode is currently supported, and it is not possible to provide a separate URL for the ruler API.
## Add or edit a Cortex or Loki managed alerting rule
@@ -26,11 +25,11 @@ This section describes the fields you fill out to create an alert.
### Alert type
- **Alert name -** Enter a descriptive name. The name will be displayed in the alert rule list, as well as added as `alertname` label to every alert instance that is created from this rule.
- **Alert type -** Select **Cortex / Loki managed alert**.
- **Data source -** Select a Prometheus or Loki data source. Only Prometheus data sources that support Cortex ruler API will be available.
- **Namespace -** Select an existing rule namespace or click **Add new** to create a new one.
- **Group -** Select an existing group within the selected namespace or click **Add new** to create a new one. Newly created rules will be added to the end of the rule group.
- **Alert name -** Enter a descriptive name. The name will be displayed in the alert rule list, as well as added as `alertname` label to every alert instance that is created from this rule.
- **Alert type -** Select **Cortex / Loki managed alert**.
- **Data source -** Select a Prometheus or Loki data source. Only Prometheus data sources that support Cortex ruler API will be available.
- **Namespace -** Select an existing rule namespace or click **Add new** to create a new one.
- **Group -** Select an existing group within the selected namespace or click **Add new** to create a new one. Newly created rules will be added to the end of the rule group.
![Alert type section screenshot](/static/img/docs/alerting/unified/rule-edit-cortex-alert-type-8-0.png 'Alert type section screenshot')
@@ -42,7 +41,7 @@ Enter a PromQL or LogQL expression. Rule will fire if evaluation result has at l
### Conditions
- **For -** For how long the selected condition should violated before an alert enters `Firing` state. When condition threshold is violated for the first time, an alert becomes `Pending`. If the **for** time elapses and the condition is still violated, it becomes `Firing`. Else it reverts back to `Normal`.
- **For -** For how long the selected condition should violated before an alert enters `Firing` state. When condition threshold is violated for the first time, an alert becomes `Pending`. If the **for** time elapses and the condition is still violated, it becomes `Firing`. Else it reverts back to `Normal`.
![Conditions section](/static/img/docs/alerting/unified/rule-edit-cortex-conditions-8-0.png 'Conditions section screenshot')
@@ -52,11 +51,11 @@ Annotations and labels can be optionally added in the details section.
#### Annotations
Annotations are key and value pairs that provide additional meta information about the alert, for example description, summary, runbook URL. They are displayed in rule and alert details in the UI and can be used in contact type message templates. Annotations can also be templated, for example `Instance {{ $labels.instance }} down` will have the evaluated `instance` label value added for every alert this rule produces.
Annotations are key and value pairs that provide additional meta information about the alert, for example description, summary, runbook URL. They are displayed in rule and alert details in the UI and can be used in contact type message templates. Annotations can also be templated, for example `Instance {{ $labels.instance }} down` will have the evaluated `instance` label value added for every alert this rule produces.
#### Labels
Labels are key value pairs that categorize or identify an alert. Labels are used to match alerts in silences or match and groups alerts in notification policies. Labels are also shown in rule or alert details in the UI and can be used in contact type message templates. For example, it is common to add a `severity` label and then configure a separate notification policy for each severity. Or one could add a `team` label and configure team specific notification policies, or silence all alerts for a particular team.
Labels are key value pairs that categorize or identify an alert. Labels are used to match alerts in silences or match and groups alerts in notification policies. Labels are also shown in rule or alert details in the UI and can be used in contact type message templates. For example, it is common to add a `severity` label and then configure a separate notification policy for each severity. Or one could add a `team` label and configure team specific notification policies, or silence all alerts for a particular team.
![Details section](/static/img/docs/alerting/unified/rule-edit-details-8-0.png 'Details section screenshot')
docs/sources/alerting/unified-alerting/alerting-rules/create-grafana-managed-rule.md
+25 -26
View File
@@ -7,7 +7,7 @@ weight = 400
# Create a Grafana managed alerting rule
Grafana allows you to create alerting rules that query one or more data sources, reduce or transform the results and compare them to each other or to fix thresholds. These rules will be executed and notifications sent by Grafana itself.
Grafana allows you to create alerting rules that query one or more data sources, reduce or transform the results and compare them to each other or to fix thresholds. These rules will be executed and notifications sent by Grafana itself.
## Add or edit a Grafana managed alerting rule
@@ -23,15 +23,15 @@ This section describes the fields you fill out to create an alert.
### Alert type
- **Alert name -** Enter a descriptive name. The name will be displayed in the alert rule list, as well as added as `alertname` label to every alert instance that is created from this rule.
- **Alert type -** Select **Grafana managed alert**.
- **Folder -** Select a folder this alert rule will belong to. To create a new folder, click on the drop down and type in a new folder name.
- **Alert name -** Enter a descriptive name. The name will be displayed in the alert rule list, as well as added as `alertname` label to every alert instance that is created from this rule.
- **Alert type -** Select **Grafana managed alert**.
- **Folder -** Select a folder this alert rule will belong to. To create a new folder, click on the drop down and type in a new folder name.
![Alert type section screenshot](/static/img/docs/alerting/unified/rule-edit-grafana-alert-type-8-0.png 'Alert type section screenshot')
### Query
Add one or more [queries]({{< relref "../../../panels/queries.md" >}}) or [expressions]({{< relref "../../../panels/expressions.md" >}}). You can use classic condition expression to create a rule that will trigger a single alert if it's threshold is met, or use reduce and math expressions to create a multi dimensional alert rule that can trigger multiple alerts, one per matching series in the query result.
Add one or more [queries]({{< relref "../../../panels/queries.md" >}}) or [expressions]({{< relref "../../../panels/expressions.md" >}}). You can use classic condition expression to create a rule that will trigger a single alert if it's threshold is met, or use reduce and math expressions to create a multi dimensional alert rule that can trigger multiple alerts, one per matching series in the query result.
#### Rule with classic condition
@@ -41,7 +41,7 @@ You can use classic condition expression to create a rule that will trigger a si
1. Add an expression. Click on **Operation** dropdown and select **Classic condition**.
1. Add one or more conditions. For each condition you can specify operator (`AND` / `OR`), aggregation function, query letter and threshold value.
If a query returns multiple series, then the aggregation function and threshold check will be evaluated for each series.It will not track alert state **per series**. This has implications that are detailed in the scenario below.
If a query returns multiple series, then the aggregation function and threshold check will be evaluated for each series.It will not track alert state **per series**. This has implications that are detailed in the scenario below.
- Alert condition with query that returns 2 series: **server1** and **server2**
- **server1** series causes the alert rule to fire and switch to state `Firing`
@@ -67,25 +67,24 @@ See or [expressions documentation]({{< relref "../../../panels/expressions.md" >
### Conditions
- **Condition -** Select the letter of the query or expression whose result will trigger the alert rule. You will likely want to select either a `classic condition` or a `math` expression.
- **Evaluate every -** How often the rule should be evaluated, executing the defined queries and expressions. Must be no less than 10 seconds and a multiple of 10 seconds. Examples: `1m`, `30s`
- **Evaluate for -** For how long the selected condition should violated before an alert enters `Alerting` state. When condition threshold is violated for the first time, an alert becomes `Pending`. If the **for** time elapses and the condition is still violated, it becomes `Alerting`. Else it reverts back to `Normal`.
- **Condition -** Select the letter of the query or expression whose result will trigger the alert rule. You will likely want to select either a `classic condition` or a `math` expression.
- **Evaluate every -** How often the rule should be evaluated, executing the defined queries and expressions. Must be no less than 10 seconds and a multiple of 10 seconds. Examples: `1m`, `30s`
- **Evaluate for -** For how long the selected condition should violated before an alert enters `Alerting` state. When condition threshold is violated for the first time, an alert becomes `Pending`. If the **for** time elapses and the condition is still violated, it becomes `Alerting`. Else it reverts back to `Normal`.
#### No Data & Error handling
Toggle **Configure no data and error handling** switch to configure how the rule should handle cases where evaluation results in error or returns no data.
| No Data Option | Description |
| --------------- | ------------------------------------------------------------------------------------------ |
| No Data | Set alert state to `NoData` and rule state to `Normal` |
| Alerting | Set alert rule state to `Alerting` |
| Ok | Set alert rule state to `Normal` |
Toggle **Configure no data and error handling** switch to configure how the rule should handle cases where evaluation results in error or returns no data.
| No Data Option | Description |
| -------------- | ------------------------------------------------------ |
| No Data | Set alert state to `NoData` and rule state to `Normal` |
| Alerting | Set alert rule state to `Alerting` |
| Ok | Set alert rule state to `Normal` |
| Error or timeout option | Description |
| ----------------------- | --------------------------------------------------- |
| Alerting | Set alert rule state to `Alerting` |
| OK | Set alert rule state to `Normal` |
| Error or timeout option | Description |
| ----------------------- | ---------------------------------- |
| Alerting | Set alert rule state to `Alerting` |
| OK | Set alert rule state to `Normal` |
![Conditions section](/static/img/docs/alerting/unified/rule-edit-grafana-conditions-8-0.png 'Conditions section screenshot')
@@ -95,11 +94,11 @@ Annotations and labels can be optionally added in the details section.
#### Annotations
Annotations are key and value pairs that provide additional meta information about the alert, for example description, summary, runbook URL. They are displayed in rule and alert details in the UI and can be used in contact type message templates. Annotations can also be templated, for example `Instance {{ $labels.instance }} down` will have the evaluated `instance` label value added for every alert this rule produces.
Annotations are key and value pairs that provide additional meta information about the alert, for example description, summary, runbook URL. They are displayed in rule and alert details in the UI and can be used in contact type message templates. Annotations can also be templated, for example `Instance {{ $labels.instance }} down` will have the evaluated `instance` label value added for every alert this rule produces.
#### Labels
Labels are key value pairs that categorize or identify an alert. Labels are used to match alerts in silences or match and groups alerts in notification policies. Labels are also shown in rule or alert details in the UI and can be used in contact type message templates. For example, it is common to add a `severity` label and then configure a separate notification policy for each severity. Or one could add a `team` label and configure team specific notification policies, or silence all alerts for a particular team. Labels can also be templated like annotations, for example `{{ $labels.namespace }}/{{ $labels.job }}` will produce a new rule label that will have the evaluated `namespace` and `job` label value added for every alert this rule produces. The rule labels take precedence over the labels produced by the query/condition.
Labels are key value pairs that categorize or identify an alert. Labels are used to match alerts in silences or match and groups alerts in notification policies. Labels are also shown in rule or alert details in the UI and can be used in contact type message templates. For example, it is common to add a `severity` label and then configure a separate notification policy for each severity. Or one could add a `team` label and configure team specific notification policies, or silence all alerts for a particular team. Labels can also be templated like annotations, for example `{{ $labels.namespace }}/{{ $labels.job }}` will produce a new rule label that will have the evaluated `namespace` and `job` label value added for every alert this rule produces. The rule labels take precedence over the labels produced by the query/condition.
![Details section](/static/img/docs/alerting/unified/rule-edit-details-8-0.png 'Details section screenshot')
@@ -107,11 +106,11 @@ Labels are key value pairs that categorize or identify an alert. Labels are use
The following template variables are available when expanding annotations and labels.
| Name | Description |
| ------- | --------------- |
| $labels | The labels from the query or condition. For example, `{{ $labels.instance }}` and `{{ $labels.job }}`. |
| Name | Description |
| ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| $labels | The labels from the query or condition. For example, `{{ $labels.instance }}` and `{{ $labels.job }}`. |
| $values | The values of all reduce and math expressions that were evaluated for this alert rule. For example, `{{ $values.A }}`, `{{ $values.A.Labels }}` and `{{ $values.A.Value }}` where `A` is the `refID` of the expression. |
| $value | The value string of the alert instance. For example, `[ var='A' labels={instance=foo} value=10 ]`. |
| $value | The value string of the alert instance. For example, `[ var='A' labels={instance=foo} value=10 ]`. |
## Preview alerts
docs/sources/alerting/unified-alerting/alerting-rules/state-and-health.md
+6
View File
@@ -9,13 +9,17 @@ keywords = ["grafana", "alerting", "guide", "state"]
The concepts of state and health for alerting rules help you understand, at a glance, several key status indicators about your alerts. Alert state, alerting rule state, and alerting rule health are related, but they each convey subtly different information.
## Alerting rule state
Indicates whether any of the timeseries resulting from evaluation of the alerting rule are in an alerting state. Alerting rule state only requires a single alerting instance to be in a pending or firing state for the alerting rule state to not be normal.
- Normal: none of the timeseries returned are in an alerting state.
- Pending: at least one of the timeseries returned are in a pending state.
- Firing: at least one of the timeseries returned are in an alerting state.
## Alert state
Alert state is an indication of the output of the alerting evaluation engine.
- Normal: the condition for the alerting rule has evaluated to **false** for every timeseries returned by the evaluation engine.
- Alerting: the condition for the alerting rule has evaluated to **true** for at least one timeseries returned by the evaluation engine and the duration, if set, **has** been met or exceeded.
- Pending: the condition for the alerting rule has evaluated to **true** for at least one timeseries returned by the evaluation engine and the duration, if set, **has not** been met or exceeded.
@@ -23,7 +27,9 @@ Alert state is an indication of the output of the alerting evaluation engine.
- Error: There was an error encountered when attempting to evaluate the alerting rule.
## Alerting rule health
Indicates the status of alerting rule evaluation.
- Ok: the rule is being evaluated, data is being returned, and no errors have been encountered.
- Error: an error was encountered when evaluating the alerting rule.
- NoData: at least one of the timeseries returned during evaluation is in a NoData state.
docs/sources/alerting/unified-alerting/contact-points.md
+26 -25
View File
@@ -17,17 +17,19 @@ Grafana alerting UI allows you to configure contact points for the Grafana manag
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. Click **Add contact point**.
1. Enter a **Name** for the contact point
1. Enter a **Name** for the contact point
1. Select contact point type and fill out mandatory fields. **Optional settings** can be expanded for more options.
1. If you'd like this contact point to notify via multiple channels, for example both email and slack, click **New contact point type** and fill out additional contact point type details.
1. Click **Save contact point** button at the bottom of the page.
## Editing a contact point
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. Find the contact point you want to edit in the contact points table and click the **pen icon** on the right side.
1. Make any changes and click **Save contact point** button at the bottom of the page.
## Deleting a contact point
## Deleting a contact point
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. Find the contact point you want to edit in the contact points table and click the **trash can icon** on the right side.
1. A confirmation dialog will open. Click **Yes, delete**.
@@ -36,31 +38,31 @@ Grafana alerting UI allows you to configure contact points for the Grafana manag
## List of notifiers supported by Grafana
Name | Type
-----|-----
[DingDing](#dingdingdingtalk) | `dingding`
[Discord](#discord) | `discord`
[Email](#email) | `email`
[Google Hangouts Chat](#google-hangouts-chat) | `googlechat`
[Kafka](#kafka) | `kafka`
Line | `line`
Microsoft Teams | `teams`
[Opsgenie](#opsgenie) | `opsgenie`
[Pagerduty](#pagerduty) | `pagerduty`
Prometheus Alertmanager | `prometheus-alertmanager`
[Pushover](#pushover) | `pushover`
Sensu | `sensu`
[Sensu Go](#sensu-go) | `sensugo`
[Slack](#slack) | `slack`
Telegram | `telegram`
Threema | `threema`
VictorOps | `victorops`
[Webhook](#webhook) | `webhook`
[Zenduty](#zenduty) | `webhook`
| Name | Type |
| --------------------------------------------- | ------------------------- |
| [DingDing](#dingdingdingtalk) | `dingding` |
| [Discord](#discord) | `discord` |
| [Email](#email) | `email` |
| [Google Hangouts Chat](#google-hangouts-chat) | `googlechat` |
| [Kafka](#kafka) | `kafka` |
| Line | `line` |
| Microsoft Teams | `teams` |
| [Opsgenie](#opsgenie) | `opsgenie` |
| [Pagerduty](#pagerduty) | `pagerduty` |
| Prometheus Alertmanager | `prometheus-alertmanager` |
| [Pushover](#pushover) | `pushover` |
| Sensu | `sensu` |
| [Sensu Go](#sensu-go) | `sensugo` |
| [Slack](#slack) | `slack` |
| Telegram | `telegram` |
| Threema | `threema` |
| VictorOps | `victorops` |
| [Webhook](#webhook) | `webhook` |
| [Zenduty](#zenduty) | `webhook` |
## Manage contact points for an external Alertmanager
Grafana alerting UI supports managing external Alertmanager configuration. Once you add an [Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page where you can select either `Grafana` or an external Alertmanager as your data source.
Grafana alerting UI supports managing external Alertmanager configuration. Once you add an [Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page where you can select either `Grafana` or an external Alertmanager as your data source.
{{< figure max-width="40%" src="/static/img/docs/alerting/unified/contact-points-select-am-8-0.gif" caption="Select Alertmanager" >}}
@@ -68,7 +70,6 @@ Grafana alerting UI supports managing external Alertmanager configuration. Once
To edit global configuration options for an alertmanager, like SMTP server that is used by default for all email contact types:
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. In the dropdown at the top of the page, select an Alertmanager data source.
1. Click **Edit global config** button at the bottom of the page.
docs/sources/alerting/unified-alerting/grafana-managed-numeric-rule.md
+22 -22
View File
@@ -28,40 +28,40 @@ If there are string columns then those columns become labels. The name of column
For a MySQL table called "DiskSpace":
| Time | Host | Disk | PercentFree
| ----------- | --- | -----| --------
| 2021-June-7 | web1 | /etc | 3
| 2021-June-7 | web2 | /var | 4
| 2021-June-7 | web3 | /var | 8
| ... | ... | ... | ...
| Time | Host | Disk | PercentFree |
| ----------- | ---- | ---- | ----------- |
| 2021-June-7 | web1 | /etc | 3 |
| 2021-June-7 | web2 | /var | 4 |
| 2021-June-7 | web3 | /var | 8 |
| ... | ... | ... | ... |
You can query the data filtering on time, but without returning the time series to Grafana. For example, an alert that would trigger per Host, Disk when there is less than 5% free space:
```sql
SELECT Host, Disk, CASE WHEN PercentFree < 5.0 THEN PercentFree ELSE 0 END FROM (
SELECT
Host,
Disk,
Avg(PercentFree)
SELECT
Host,
Disk,
Avg(PercentFree)
FROM DiskSpace
Group By
Host,
Group By
Host,
Disk
Where __timeFilter(Time)
```
This query returns the following Table response to Grafana:
| Host | Disk | PercentFree
| --- | -----| --------
| web1 | /etc | 3
| web2 | /var | 4
| web3 | /var | 0
| Host | Disk | PercentFree |
| ---- | ---- | ----------- |
| web1 | /etc | 3 |
| web2 | /var | 4 |
| web3 | /var | 0 |
When this query is used as the **condition** in an alert rule, then the non-zero will be alerting. As a result, three alert instances are produced:
| Labels | Status
| ----------------------| ------
| {Host=web1,disk=/etc} | Alerting
| {Host=web2,disk=/var} | Alerting
| {Host=web3,disk=/var} | Normal
| Labels | Status |
| --------------------- | -------- |
| {Host=web1,disk=/etc} | Alerting |
| {Host=web2,disk=/var} | Alerting |
| {Host=web3,disk=/var} | Normal |
docs/sources/alerting/unified-alerting/message-templating/_index.md
+10 -7
View File
@@ -25,6 +25,7 @@ Grafana alerting UI allows you to configure templates for the Grafana managed al
> **Note:** Currently the configuration of the embedded Alertmanager is shared across organisations. Therefore users are advised to use the new Grafana 8 Alerts only if they have one organisation otherwise templates for the Grafana managed alerts will be visible by all organizations
### Create a template
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. Click **Add template**.
1. Fill in **Name** and **Content** fields.
@@ -32,18 +33,19 @@ Grafana alerting UI allows you to configure templates for the Grafana managed al
**Note** The template name used to reference this template in templating is not the value of the **Name** field, but the parameter to `define` tag in the content. When creating a template you can omit `define` entirely and it will be added automatically with same value as **Name** field. It's recommended to use the same name for `define` and **Name** field to avoid confusion.
<img src="/static/img/docs/alerting/unified/templates-create-8-0.png" width="600px">
### Edit a template
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. Find the template you want to edit in the templates table and click the **pen icon** on the right side.
1. Make any changes and click **Save template** button at the bottom of the page.
### Delete a template
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. Find the template you want to edit in the templates table and click the **trash can icon** on the right side.
1. A confirmation dialog will open. Click **Yes, delete**.
1. A confirmation dialog will open. Click **Yes, delete**.
**Note** You are not prevented from deleting templates that are in use somewhere in contact points or other templates. Be careful!
@@ -51,23 +53,23 @@ Grafana alerting UI allows you to configure templates for the Grafana managed al
To use a template:
Enter `{{ template "templatename" . }}` into a contact point field, where `templatename` is the `define` parameter of a template.
Enter `{{ template "templatename" . }}` into a contact point field, where `templatename` is the `define` parameter of a template.
<img src="/static/img/docs/alerting/unified/contact-points-use-template-8-0.png" width="600px">
### Template examples
Here is an example of a template to render a single alert:
```
{{ define "alert" }}
[{{.Status}}] {{ .Labels.alertname }}
Labels:
{{ range .Labels.SortedPairs }}
{{ .Name }}: {{ .Value }}
{{ end }}
{{ if gt (len .Annotations) 0 }}
Annotations:
{{ range .Annotations.SortedPairs }}
@@ -85,6 +87,7 @@ Here is an example of a template to render a single alert:
```
Template to render entire notification message:
```
{{ define "message" }}
{{ if gt (len .Alerts.Firing) 0 }}
@@ -100,6 +103,6 @@ Template to render entire notification message:
## Manage templates for an external Alertmanager
Grafana alerting UI supports managing external Alertmanager configuration. Once you add an [Alertmanager data source]({{< relref "../../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page, allowing you to select either `Grafana` or an external Alertmanager data source.
Grafana alerting UI supports managing external Alertmanager configuration. Once you add an [Alertmanager data source]({{< relref "../../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page, allowing you to select either `Grafana` or an external Alertmanager data source.
{{< figure max-width="40%" src="/static/img/docs/alerting/unified/contact-points-select-am-8-0.gif" caption="Select Alertmanager" >}}
docs/sources/alerting/unified-alerting/message-templating/template-data.md
+39 -40
View File
@@ -3,41 +3,40 @@ title = "Template data"
keywords = ["grafana", "alerting", "guide", "contact point", "templating"]
+++
# Template data
Template data is passed on to [message templates]({{< relref "./_index.md" >}}) as well as sent as payload to webhook pushes.
Name | Type | Notes
------------------|----------|-----------------------------------------------------------------
Receiver | string | Name of the contact point that the notification is being sent to.
Status | string | `firing` if at least one alert is firing, otherwise `resolved`.
Alerts | Alert | List of alert objects that are included in this notification (see below).
GroupLabels | KeyValue | Labels these alerts were grouped by.
CommonLabels | KeyValue | Labels common to all the alerts included in this notification.
CommonAnnotations | KeyValue | Annotations common to all the alerts included in this notification.
ExternalURL | string | Back link to the Grafana that sent the notification. If using external Alertmanager, back link to this Alertmanager.
| Name | Type | Notes |
| ----------------- | -------- | -------------------------------------------------------------------------------------------------------------------- |
| Receiver | string | Name of the contact point that the notification is being sent to. |
| Status | string | `firing` if at least one alert is firing, otherwise `resolved`. |
| Alerts | Alert | List of alert objects that are included in this notification (see below). |
| GroupLabels | KeyValue | Labels these alerts were grouped by. |
| CommonLabels | KeyValue | Labels common to all the alerts included in this notification. |
| CommonAnnotations | KeyValue | Annotations common to all the alerts included in this notification. |
| ExternalURL | string | Back link to the Grafana that sent the notification. If using external Alertmanager, back link to this Alertmanager. |
The `Alerts` type exposes functions for filtering alerts:
* `Alerts.Firing` returns a list of firing alerts.
* `Alerts.Resolved` returns a list of resolved alerts.
- `Alerts.Firing` returns a list of firing alerts.
- `Alerts.Resolved` returns a list of resolved alerts.
## Alert
Name | Type | Notes
-------------|-----------|---------------------------------------------------------------------
Status | string | `firing` or `resolved`.
Labels | KeyValue | A set of labels attached to the alert.
Annotations | KeyValue | A set of annotations attached to the alert.
StartsAt | time.Time | Time the alert started firing.
EndsAt | time.Time | Only set if the end time of an alert is known. Otherwise set to a configurable timeout period from the time since the last alert was received.
GeneratorURL | string | A back link to Grafana or external Alertmanager.
SilenceURL | string | Link to grafana silence for with labels for this alert pre-filled. Only for Grafana managed alerts.
DashboardURL | string | Link to grafana dashboard, if alert rule belongs to one. Only for Grafana managed alerts.
PanelURL | string | Link to grafana dashboard panel, if alert rule belongs to one. Only for Grafana managed alerts.
Fingerprint | string | Fingerprint that can be used to identify the alert.
ValueString | string | A string that contains the labels and value of each reduced expression in the alert.
| Name | Type | Notes |
| ------------ | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Status | string | `firing` or `resolved`. |
| Labels | KeyValue | A set of labels attached to the alert. |
| Annotations | KeyValue | A set of annotations attached to the alert. |
| StartsAt | time.Time | Time the alert started firing. |
| EndsAt | time.Time | Only set if the end time of an alert is known. Otherwise set to a configurable timeout period from the time since the last alert was received. |
| GeneratorURL | string | A back link to Grafana or external Alertmanager. |
| SilenceURL | string | Link to grafana silence for with labels for this alert pre-filled. Only for Grafana managed alerts. |
| DashboardURL | string | Link to grafana dashboard, if alert rule belongs to one. Only for Grafana managed alerts. |
| PanelURL | string | Link to grafana dashboard panel, if alert rule belongs to one. Only for Grafana managed alerts. |
| Fingerprint | string | Fingerprint that can be used to identify the alert. |
| ValueString | string | A string that contains the labels and value of each reduced expression in the alert. |
## KeyValue
@@ -54,23 +53,23 @@ Here is an example containing two annotations:
In addition to direct access of data (labels and annotations) stored as KeyValue, there are also methods for sorting, removing and transforming.
Name | Arguments | Returns | Notes
------------|-----------|-----------------------------------------|----------------
SortedPairs | | Sorted list of key & value string pairs |
Remove | []string | KeyValue | Returns a copy of the Key/Value map without the given keys.
Names | | []string | List of label names
Values | | []string | List of label values
| Name | Arguments | Returns | Notes |
| ----------- | --------- | --------------------------------------- | ----------------------------------------------------------- |
| SortedPairs | | Sorted list of key & value string pairs |
| Remove | []string | KeyValue | Returns a copy of the Key/Value map without the given keys. |
| Names | | []string | List of label names |
| Values | | []string | List of label values |
## Functions
Some functions to transform values are also available, along with [default functions provided by Go templating](https://golang.org/pkg/text/template/#hdr-Functions).
Name | Arguments | Returns
-------------|------------------------------|----------------------------------------------
title | string | Capitalizes first character of each word.
toUpper | string | Converts all characters to upper case.
match | pattern, string | Match a string using RegExp.
reReplaceAll | pattern, replacement, string | RegExp substitution, unanchored.
join | string, []string | Concatenates the elements of the second argument to create a single string. First argument is the separator.
safeHtml | string | Marks string as HTML, not requiring auto-escaping.
stringSlice | ...string | Returns passed strings as slice of strings.
| Name | Arguments | Returns |
| ------------ | ---------------------------- | ------------------------------------------------------------------------------------------------------------ |
| title | string | Capitalizes first character of each word. |
| toUpper | string | Converts all characters to upper case. |
| match | pattern, string | Match a string using RegExp. |
| reReplaceAll | pattern, replacement, string | RegExp substitution, unanchored. |
| join | string, []string | Concatenates the elements of the second argument to create a single string. First argument is the separator. |
| safeHtml | string | Marks string as HTML, not requiring auto-escaping. |
| stringSlice | ...string | Returns passed strings as slice of strings. |
docs/sources/alerting/unified-alerting/notification-policies.md
+9 -13
View File
@@ -20,7 +20,7 @@ To access notification policy editing page, In the Grafana side bar, hover your
### Edit root notification policy
1. Click **edit** button on the top right of the root policy box.
1. Make changes and click **save** button.
1. Make changes and click **save** button.
### Add new specific policy
@@ -32,7 +32,6 @@ To add a nested policy to an existing specific policy, expand the parent policy
To edit a specific policy, find it in the specific routing table and click **Edit** button. Make your changes and click **Save policy**.
### Root policy fields
- **Default contact point -** The [contact point]({{< relref "./contact-points.md" >}}) to send notifications to that did not match any specific policy.
@@ -44,16 +43,14 @@ Group timing options
- **Group interval -** - How long to wait before sending an notification when an alert has been added to a group for which there has already been a notification. Default is 5 minutes.
- **Repeat interval -** - How long to wait before re-sending a notification after one has already been sent and no new alerts were added to the group. Default is 4 hours.
### Specific policy fields
- **Contact point -** The [contact point]({{< relref "./contact-points.md" >}}) to send notification to if alert matched this specific policy but did not match any of it's nested policies, or there were no nested specific policies.
- **Matching labels -** Rules for matching alert labels. See ["How label matching works"](#how-label-matching-works) below for details.
- **Continue matching subsequent sibling nodes -** If not enabled and an alert matches this policy but not any of it's nested policies, matching will stop and a notification will be sent to the contact point defined on this policy. If enabled, notification will be sent but alert will continue matching subsequent siblings of this policy, thus sending more than one notification. Use this if for example you want to send notification to a catch-all contact point as well as to one of more specific contact points handled by subsequent policies.
- **Override grouping** - Toggle if you want to override grouping for this policy. If toggled, you will be able to specify grouping same as for root policy described above. If not toggled, root policy grouping will be used.
- **Continue matching subsequent sibling nodes -** If not enabled and an alert matches this policy but not any of it's nested policies, matching will stop and a notification will be sent to the contact point defined on this policy. If enabled, notification will be sent but alert will continue matching subsequent siblings of this policy, thus sending more than one notification. Use this if for example you want to send notification to a catch-all contact point as well as to one of more specific contact points handled by subsequent policies.
- **Override grouping** - Toggle if you want to override grouping for this policy. If toggled, you will be able to specify grouping same as for root policy described above. If not toggled, root policy grouping will be used.
- **Override group timings** Toggle if you want to override group timings for this policy. If toggled, you will be able to specify group timings same as for root policy described above. If not toggled, root policy group timings will be used.
### How label matching works
A policy will match an alert if alert's labels match all of the "Matching Labels" specified on the policy.
@@ -63,15 +60,14 @@ A policy will match an alert if alert's labels match all of the "Matching Labels
- The **Regex** checkbox specifies if the inputted **Value** should be matched against labels as a regular expression. The regular expression is always anchored. If not selected it is an exact string match.
- The **Equal** checkbox specifies if the match should include alert instances that match or do not match. If not checked, the silence includes alert instances _do not_ match.
## Example setup
One usage example would be:
* Create a "default" contact point for most alerts with a non invasive contact point type, like a slack message, and set it on root policy
* Edit root policy grouping to group alerts by `cluster`, `namespace` and `alertname` so you get a notification per alert rule and specific k8s cluster & namespace.
* Create specific route for alerts coming from development cluster with an appropriate contact point
* Create a specific route for alerts with "critical" severity with a more invasive contact point type, like pager duty notification
* Create specific routes for particular teams that handle their own onduty rotations
- Create a "default" contact point for most alerts with a non invasive contact point type, like a slack message, and set it on root policy
- Edit root policy grouping to group alerts by `cluster`, `namespace` and `alertname` so you get a notification per alert rule and specific k8s cluster & namespace.
- Create specific route for alerts coming from development cluster with an appropriate contact point
- Create a specific route for alerts with "critical" severity with a more invasive contact point type, like pager duty notification
- Create specific routes for particular teams that handle their own onduty rotations
![Notification policies screenshot](/static/img/docs/alerting/unified/notification-policies-8-0.png 'Notification policies screenshot')
docs/sources/alerting/unified-alerting/opt-in.md
+5 -4
View File
@@ -8,11 +8,11 @@ weight = 128
Setting the `ngalert` feature toggle enables the new Grafana 8 alerting system.
>**Note:** We recommend that you backup Grafana's database before enabling this feature. If you are using PostgreSQL as the backend data source, then the minimum required version is 9.5.
> **Note:** We recommend that you backup Grafana's database before enabling this feature. If you are using PostgreSQL as the backend data source, then the minimum required version is 9.5.
At startup, when [the feature toggle is enabled]({{< relref "../../administration/configuration.md">}}#feature_toggles), the legacy Grafana dashboard alerting is disabled and existing dashboard alerts are migrated into a format that is compatible with the Grafana 8 alerting system. You can view these migrated rules, alongside any new alerts you create after the migration, from the Alerting page of your Grafana instance.
>**Note:** Since the new system stores the notification log and silences on disk, we require the use of persistent disks for using Grafana 8 alerts. Otherwise, the silences and notification log will get lost on a restart, and you might get unwanted or duplicate notifications.
> **Note:** Since the new system stores the notification log and silences on disk, we require the use of persistent disks for using Grafana 8 alerts. Otherwise, the silences and notification log will get lost on a restart, and you might get unwanted or duplicate notifications.
Read and write access to dashboard alerts in Grafana versions 7 and earlier were governed by the dashboard and folder permissions under which the alerts were stored. In Grafana 8, alerts are stored in folders and inherit the permissions of those folders. During the migration, dashboard alert permissions are matched to the new rules permissions as follows:
@@ -20,7 +20,7 @@ Read and write access to dashboard alerts in Grafana versions 7 and earlier were
- If there are no dashboard permissions and the dashboard is under a folder, then the rule is linked to this folder and inherits its permissions.
- If there are no dashboard permissions and the dashboard is under the General folder, then the rule is linked to the `General Alerting` folder and the rule inherits the default permissions.
During beta, Grafana 8 alerting system can retrieve rules from all available Prometheus, Loki, and Alertmanager data sources. It might not be able to fetch rules from all other supported data sources at this time.
During beta, Grafana 8 alerting system can retrieve rules from all available Prometheus, Loki, and Alertmanager data sources. It might not be able to fetch rules from all other supported data sources at this time.
Also notification channels are migrated to an Alertmanager configuration with the appropriate routes and receivers. Default notification channels are added as contact points to the default route. Notification channels not associated with any Dashboard alert go to the `autogen-unlinked-channel-recv` route.
@@ -28,6 +28,7 @@ Since `Hipchat` and `Sensu` are discontinued, they are not migrated to the new a
Finally, silences (expiring after one year) are created for all paused dashboard alerts.
## Disabling Grafana 8 Alerting after migration
To disable Grafana 8 Alerting, remove or disable the `ngalert` feature toggle. Dashboard alerts will be re-enabled and any alerts created during or after the migration are deleted.
>**Note:** Any alerting rules created in the Grafana 8 Alerting system will be lost when migrating back to dashboard alerts
> **Note:** Any alerting rules created in the Grafana 8 Alerting system will be lost when migrating back to dashboard alerts
docs/sources/alerting/unified-alerting/silences.md
+3 -3
View File
@@ -23,7 +23,7 @@ To add a silence:
1. Click the **New Silence** button.
1. Select the start and end date in **Silence start and end** to indicate when the silence should go into effect and expire.
1. Optionally, update the **Duration** to alter the time for the end of silence in the previous step to correspond to the start plus the duration.
1. Enter one or more *Matching Labels* by filling out the **Name** and **Value** fields. Matchers determine which rules the silence will apply to.
1. Enter one or more _Matching Labels_ by filling out the **Name** and **Value** fields. Matchers determine which rules the silence will apply to.
1. Enter a **Comment**.
1. Enter the name of the owner in **Creator**.
1. Click **Create**.
@@ -46,12 +46,12 @@ Alert instances that have labels that match all of the "Matching Labels" specifi
## Manage silences for an external Alertmanager
Grafana alerting UI supports managing external Alertmanager silences. Once you add an [Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page where you can select either `Grafana` or an external Alertmanager as your data source.
Grafana alerting UI supports managing external Alertmanager silences. Once you add an [Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page where you can select either `Grafana` or an external Alertmanager as your data source.
## Create a URL to silence form with defaults filled in
When linking to silence form, you can provide default matching labels and comment via `matchers` and `comment` query parameters. `matchers` expects one more matching labels of type `[label][operator][value]` joined by a comma. `operator` can be one of `=` (equals, not regex), `!=` (not equals, not regex), `=~` (equals, regex), `!~` (not equals, not regex).
For example, to link to silence form with matching labels `severity=critical` & `cluster!~europe-.*` and comment `Silence critical EU alerts`, create a URL `https://mygrafana/aleting/silence/new?matchers=severity%3Dcritical%2Ccluster!~europe-*&comment=Silence%20critical%20EU%20alert`.
For example, to link to silence form with matching labels `severity=critical` & `cluster!~europe-.*` and comment `Silence critical EU alerts`, create a URL `https://mygrafana/aleting/silence/new?matchers=severity%3Dcritical%2Ccluster!~europe-*&comment=Silence%20critical%20EU%20alert`.
To link to a new silence page for an [external Alertmanager]({{< relref "../../datasources/alertmanager.md" >}}), add a `alertmanager` query parameter with the Alertmanager data source name.