public/grafana
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #12145 from grafana/alerting_reminder

Browse Source 
Alerting notification reminder
This commit is contained in:
Marcus Efraimsson
2018-09-03 14:41:38 +02:00
committed by GitHub
parent f8c2b23c86 6995242b8b
commit 1ceca5d8ec
34 changed files with 810 additions and 200 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sources/alerting/notifications.md
+44 -21
View File
@@ -16,12 +16,11 @@ weight = 2
When an alert changes state, it sends out notifications. Each alert rule can have
multiple notifications. In order to add a notification to an alert rule you first need
to add and configure a `notification` channel (can be email, PagerDuty or other integration). This is done from the Notification Channels page.
to add and configure a `notification` channel (can be email, PagerDuty or other integration).
This is done from the Notification Channels page.
## Notification Channel Setup
{{< imgbox max-width="30%" img="/img/docs/v50/alerts_notifications_menu.png" caption="Alerting Notification Channels" >}}
On the Notification Channels page hit the `New Channel` button to go the page where you
can configure and setup a new Notification Channel.
@@ -30,7 +29,31 @@ sure it's setup correctly.
### Send on all alerts
When checked, this option will nofity for all alert rules - existing and new.
When checked, this option will notify for all alert rules - existing and new.
### Send reminders
> Only available in Grafana v5.3 and above.
{{< docs-imagebox max-width="600px" img="/img/docs/v53/alerting_notification_reminders.png" class="docs-image--right" caption="Alerting notification reminders setup" >}}
When this option is checked additional notifications (reminders) will be sent for triggered alerts. You can specify how often reminders
should be sent using number of seconds (s), minutes (m) or hours (h), for example `30s`, `3m`, `5m` or `1h` etc.
**Important:** Alert reminders are sent after rules are evaluated. Therefore a reminder can never be sent more frequently than a configured [alert rule evaluation interval](/alerting/rules/#name-evaluation-interval).
These examples show how often and when reminders are sent for a triggered alert.
Alert rule evaluation interval | Send reminders every | Reminder sent every (after last alert notification)
---------- | ----------- | -----------
`30s` | `15s` | ~30 seconds
`1m` | `5m` | ~5 minutes
`5m` | `15m` | ~15 minutes
`6m` | `20m` | ~24 minutes
`1h` | `15m` | ~1 hour
`1h` | `2h` | ~2 hours
<div class="clearfix"></div>
## Supported Notification Types
@@ -132,23 +155,23 @@ Once these two properties are set, you can send the alerts to Kafka for further
### All supported notifiers
Name | Type |Support images
-----|------------ | ------
Slack | `slack` | yes
Pagerduty | `pagerduty` | yes
Email | `email` | yes
Webhook | `webhook` | link
Kafka | `kafka` | no
Hipchat | `hipchat` | yes
VictorOps | `victorops` | yes
Sensu | `sensu` | yes
OpsGenie | `opsgenie` | yes
Threema | `threema` | yes
Pushover | `pushover` | no
Telegram | `telegram` | no
Line | `line` | no
Prometheus Alertmanager | `prometheus-alertmanager` | no
Microsoft Teams | `teams` | yes
Name | Type |Support images | Support reminders
-----|------------ | ------ | ------ |
Slack | `slack` | yes | yes
Pagerduty | `pagerduty` | yes | yes
Email | `email` | yes | yes
Webhook | `webhook` | link | yes
Kafka | `kafka` | no | yes
Hipchat | `hipchat` | yes | yes
VictorOps | `victorops` | yes | yes
Sensu | `sensu` | yes | yes
OpsGenie | `opsgenie` | yes | yes
Threema | `threema` | yes | yes
Pushover | `pushover` | no | yes
Telegram | `telegram` | no | yes
Line | `line` | no | yes
Microsoft Teams | `teams` | yes | yes
Prometheus Alertmanager | `prometheus-alertmanager` | no | no
docs/sources/alerting/rules.md
+5
View File
@@ -88,6 +88,11 @@ So as you can see from the above scenario Grafana will not send out notification
to fire if the rule already is in state `Alerting`. To improve support for queries that return multiple series
we plan to track state **per series** in a future release.
> Starting with Grafana v5.3 you can configure reminders to be sent for triggered alerts. This will send additional notifications
> when an alert continues to fire. If other series (like server2 in the example above) also cause the alert rule to fire they will
> be included in the reminder notification. Depending on what notification channel you're using you may be able to take advantage
> of this feature for identifying new/existing series causing alert to fire. [Read more about notification reminders here](/alerting/notifications/#send-reminders).
### No Data / Null values
Below your conditions you can configure how the rule evaluation engine should handle queries that return no data or only null values.
docs/sources/http_api/alerting.md
+36 -12
View File
@@ -50,6 +50,7 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
```http
HTTP/1.1 200
Content-Type: application/json
[
{
"id": 1,
@@ -86,6 +87,7 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
```http
HTTP/1.1 200
Content-Type: application/json
{
"id": 1,
"dashboardId": 1,
@@ -146,6 +148,7 @@ JSON Body Schema:
```http
HTTP/1.1 200
Content-Type: application/json
{
"alertId": 1,
"state": "Paused",
@@ -177,6 +180,7 @@ JSON Body Schema:
```http
HTTP/1.1 200
Content-Type: application/json
{
"state": "Paused",
"message": "alert paused",
@@ -204,14 +208,21 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
HTTP/1.1 200
Content-Type: application/json
{
"id": 1,
"name": "Team A",
"type": "email",
"isDefault": true,
"created": "2017-01-01 12:45",
"updated": "2017-01-01 12:45"
}
[
{
"id": 1,
"name": "Team A",
"type": "email",
"isDefault": false,
"sendReminder": false,
"settings": {
"addresses": "carl@grafana.com;dev@grafana.com"
},
"created": "2018-04-23T14:44:09+02:00",
"updated": "2018-08-20T15:47:49+02:00"
}
]
```
## Create alert notification
@@ -232,6 +243,7 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
"name": "new alert notification", //Required
"type": "email", //Required
"isDefault": false,
"sendReminder": false,
"settings": {
"addresses": "carl@grafana.com;dev@grafana.com"
}
@@ -243,14 +255,18 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
```http
HTTP/1.1 200
Content-Type: application/json
{
"id": 1,
"name": "new alert notification",
"type": "email",
"isDefault": false,
"settings": { addresses: "carl@grafana.com;dev@grafana.com"} }
"created": "2017-01-01 12:34",
"updated": "2017-01-01 12:34"
"sendReminder": false,
"settings": {
"addresses": "carl@grafana.com;dev@grafana.com"
},
"created": "2018-04-23T14:44:09+02:00",
"updated": "2018-08-20T15:47:49+02:00"
}
```
@@ -271,6 +287,8 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
"name": "new alert notification", //Required
"type": "email", //Required
"isDefault": false,
"sendReminder": true,
"frequency": "15m",
"settings": {
"addresses: "carl@grafana.com;dev@grafana.com"
}
@@ -282,12 +300,17 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
```http
HTTP/1.1 200
Content-Type: application/json
{
"id": 1,
"name": "new alert notification",
"type": "email",
"isDefault": false,
"settings": { addresses: "carl@grafana.com;dev@grafana.com"} }
"sendReminder": true,
"frequency": "15m",
"settings": {
"addresses": "carl@grafana.com;dev@grafana.com"
},
"created": "2017-01-01 12:34",
"updated": "2017-01-01 12:34"
}
@@ -311,6 +334,7 @@ Authorization: Bearer eyJrIjoiT0tTcG1pUlY2RnVKZTFVaDFsNFZXdE9ZWmNrMkZYbk
```http
HTTP/1.1 200
Content-Type: application/json
{
"message": "Notification deleted"
}