|
|
|
@@ -37,41 +37,41 @@ This is done from the Notification channels page.
|
|
|
|
|
|
|
|
|
|
These examples show how often and when reminders are sent for a triggered alert.
|
|
|
|
|
|
|
|
|
|
Alert rule evaluation interval | Send reminders every | Reminder sent every (after last alert notification)
|
|
|
|
|
---------- | ----------- | -----------
|
|
|
|
|
`30s` | `15s` | ~30 seconds
|
|
|
|
|
`1m` | `5m` | ~5 minutes
|
|
|
|
|
`5m` | `15m` | ~15 minutes
|
|
|
|
|
`6m` | `20m` | ~24 minutes
|
|
|
|
|
`1h` | `15m` | ~1 hour
|
|
|
|
|
`1h` | `2h` | ~2 hours
|
|
|
|
|
| Alert rule evaluation interval | Send reminders every | Reminder sent every (after last alert notification) |
|
|
|
|
|
| ------------------------------ | -------------------- | --------------------------------------------------- |
|
|
|
|
|
| `30s` | `15s` | ~30 seconds |
|
|
|
|
|
| `1m` | `5m` | ~5 minutes |
|
|
|
|
|
| `5m` | `15m` | ~15 minutes |
|
|
|
|
|
| `6m` | `20m` | ~24 minutes |
|
|
|
|
|
| `1h` | `15m` | ~1 hour |
|
|
|
|
|
| `1h` | `2h` | ~2 hours |
|
|
|
|
|
|
|
|
|
|
<div class="clearfix"></div>
|
|
|
|
|
|
|
|
|
|
## List of supported notifiers
|
|
|
|
|
|
|
|
|
|
Name | Type | Supports images | Support alert rule tags
|
|
|
|
|
-----|------|---------------- | -----------------------
|
|
|
|
|
[DingDing](#dingdingdingtalk) | `dingding` | yes, external only | no
|
|
|
|
|
[Discord](#discord) | `discord` | yes | no
|
|
|
|
|
[Email](#email) | `email` | yes | no
|
|
|
|
|
[Google Hangouts Chat](#google-hangouts-chat) | `googlechat` | yes, external only | no
|
|
|
|
|
Hipchat | `hipchat` | yes, external only | no
|
|
|
|
|
[Kafka](#kafka) | `kafka` | yes, external only | no
|
|
|
|
|
Line | `line` | yes, external only | no
|
|
|
|
|
Microsoft Teams | `teams` | yes, external only | no
|
|
|
|
|
[Opsgenie](#opsgenie) | `opsgenie` | yes, external only | yes
|
|
|
|
|
[Pagerduty](#pagerduty) | `pagerduty` | yes, external only | yes
|
|
|
|
|
Prometheus Alertmanager | `prometheus-alertmanager` | yes, external only | yes
|
|
|
|
|
[Pushover](#pushover) | `pushover` | yes | no
|
|
|
|
|
Sensu | `sensu` | yes, external only | no
|
|
|
|
|
[Sensu Go](#sensu-go) | `sensugo` | yes, external only | no
|
|
|
|
|
[Slack](#slack) | `slack` | yes | no
|
|
|
|
|
Telegram | `telegram` | yes | no
|
|
|
|
|
Threema | `threema` | yes, external only | no
|
|
|
|
|
VictorOps | `victorops` | yes, external only | yes
|
|
|
|
|
[Webhook](#webhook) | `webhook` | yes, external only | yes
|
|
|
|
|
[Zenduty](#zenduty) | `webhook` | yes, external only | yes
|
|
|
|
|
| Name | Type | Supports images | Support alert rule tags |
|
|
|
|
|
| --------------------------------------------- | ------------------------- | ------------------ | ----------------------- |
|
|
|
|
|
| [DingDing](#dingdingdingtalk) | `dingding` | yes, external only | no |
|
|
|
|
|
| [Discord](#discord) | `discord` | yes | no |
|
|
|
|
|
| [Email](#email) | `email` | yes | no |
|
|
|
|
|
| [Google Hangouts Chat](#google-hangouts-chat) | `googlechat` | yes, external only | no |
|
|
|
|
|
| Hipchat | `hipchat` | yes, external only | no |
|
|
|
|
|
| [Kafka](#kafka) | `kafka` | yes, external only | no |
|
|
|
|
|
| Line | `line` | yes, external only | no |
|
|
|
|
|
| Microsoft Teams | `teams` | yes, external only | no |
|
|
|
|
|
| [Opsgenie](#opsgenie) | `opsgenie` | yes, external only | yes |
|
|
|
|
|
| [Pagerduty](#pagerduty) | `pagerduty` | yes, external only | yes |
|
|
|
|
|
| Prometheus Alertmanager | `prometheus-alertmanager` | yes, external only | yes |
|
|
|
|
|
| [Pushover](#pushover) | `pushover` | yes | no |
|
|
|
|
|
| Sensu | `sensu` | yes, external only | no |
|
|
|
|
|
| [Sensu Go](#sensu-go) | `sensugo` | yes, external only | no |
|
|
|
|
|
| [Slack](#slack) | `slack` | yes | no |
|
|
|
|
|
| Telegram | `telegram` | yes | no |
|
|
|
|
|
| Threema | `threema` | yes, external only | no |
|
|
|
|
|
| VictorOps | `victorops` | yes, external only | yes |
|
|
|
|
|
| [Webhook](#webhook) | `webhook` | yes, external only | yes |
|
|
|
|
|
| [Zenduty](#zenduty) | `webhook` | yes, external only | yes |
|
|
|
|
|
|
|
|
|
|
### Email
|
|
|
|
|
|
|
|
|
@@ -83,10 +83,10 @@ able to access the image.
|
|
|
|
|
|
|
|
|
|
> **Note:** Template variables are not supported in email alerts.
|
|
|
|
|
|
|
|
|
|
Setting | Description
|
|
|
|
|
---------- | -----------
|
|
|
|
|
Single email | Send a single email to all recipients. Disabled per default.
|
|
|
|
|
Addresses | Email addresses to recipients. You can enter multiple email addresses using a ";" separator.
|
|
|
|
|
| Setting | Description |
|
|
|
|
|
| ------------ | -------------------------------------------------------------------------------------------- |
|
|
|
|
|
| Single email | Send a single email to all recipients. Disabled per default. |
|
|
|
|
|
| Addresses | Email addresses to recipients. You can enter multiple email addresses using a ";" separator. |
|
|
|
|
|
|
|
|
|
|
### Slack
|
|
|
|
|
|
|
|
|
@@ -98,17 +98,17 @@ firing alerts in the Slack messages you have to configure either the [external i
|
|
|
|
|
in Grafana or a bot integration via Slack Apps. [Follow Slack's guide to set up a bot integration](https://api.slack.com/bot-users) and use the token
|
|
|
|
|
provided, which starts with "xoxb".
|
|
|
|
|
|
|
|
|
|
Setting | Description
|
|
|
|
|
---------- | -----------
|
|
|
|
|
Url | Slack incoming webhook URL, or eventually the [chat.postMessage](https://api.slack.com/methods/chat.postMessage) Slack API endpoint.
|
|
|
|
|
Username | Set the username for the bot's message.
|
|
|
|
|
Recipient | Allows you to override the Slack recipient. You must either provide a channel Slack ID, a user Slack ID, a username reference (@<user>, all lowercase, no whitespace), or a channel reference (#<channel>, all lowercase, no whitespace). If you use the `chat.postMessage` Slack API endpoint, this is required.
|
|
|
|
|
Icon emoji | Provide an emoji to use as the icon for the bot's message. Ex :smile:
|
|
|
|
|
Icon URL | Provide a URL to an image to use as the icon for the bot's message.
|
|
|
|
|
Mention Users | Optionally mention one or more users in the Slack notification sent by Grafana. You have to refer to users, comma-separated, via their corresponding Slack IDs (which you can find by clicking the overflow button on each user's Slack profile).
|
|
|
|
|
Mention Groups | Optionally mention one or more groups in the Slack notification sent by Grafana. You have to refer to groups, comma-separated, via their corresponding Slack IDs (which you can get from each group's Slack profile URL).
|
|
|
|
|
Mention Channel | Optionally mention either all channel members or just active ones.
|
|
|
|
|
Token | If provided, Grafana will upload the generated image via Slack's file.upload API method, not the external image destination. If you use the `chat.postMessage` Slack API endpoint, this is required.
|
|
|
|
|
| Setting | Description |
|
|
|
|
|
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
|
|
|
| Url | Slack incoming webhook URL, or eventually the [chat.postMessage](https://api.slack.com/methods/chat.postMessage) Slack API endpoint. |
|
|
|
|
|
| Username | Set the username for the bot's message. |
|
|
|
|
|
| Recipient | Allows you to override the Slack recipient. You must either provide a channel Slack ID, a user Slack ID, a username reference (@<user>, all lowercase, no whitespace), or a channel reference (#<channel>, all lowercase, no whitespace). If you use the `chat.postMessage` Slack API endpoint, this is required. |
|
|
|
|
|
| Icon emoji | Provide an emoji to use as the icon for the bot's message. Ex :smile: |
|
|
|
|
|
| Icon URL | Provide a URL to an image to use as the icon for the bot's message. |
|
|
|
|
|
| Mention Users | Optionally mention one or more users in the Slack notification sent by Grafana. You have to refer to users, comma-separated, via their corresponding Slack IDs (which you can find by clicking the overflow button on each user's Slack profile). |
|
|
|
|
|
| Mention Groups | Optionally mention one or more groups in the Slack notification sent by Grafana. You have to refer to groups, comma-separated, via their corresponding Slack IDs (which you can get from each group's Slack profile URL). |
|
|
|
|
|
| Mention Channel | Optionally mention either all channel members or just active ones. |
|
|
|
|
|
| Token | If provided, Grafana will upload the generated image via Slack's file.upload API method, not the external image destination. If you use the `chat.postMessage` Slack API endpoint, this is required. |
|
|
|
|
|
|
|
|
|
|
If you are using the token for a slack bot, then you have to invite the bot to the channel you want to send notifications and add the channel to the recipient field.
|
|
|
|
|
|
|
|
|
@@ -116,12 +116,12 @@ If you are using the token for a slack bot, then you have to invite the bot to t
|
|
|
|
|
|
|
|
|
|
To setup Opsgenie you will need an API Key and the Alert API Url. These can be obtained by configuring a new [Grafana Integration](https://docs.opsgenie.com/docs/grafana-integration).
|
|
|
|
|
|
|
|
|
|
Setting | Description
|
|
|
|
|
--------|------------
|
|
|
|
|
Alert API URL | The API URL for your Opsgenie instance. This will normally be either `https://api.opsgenie.com` or, for EU customers, `https://api.eu.opsgenie.com`.
|
|
|
|
|
API Key | The API Key as provided by Opsgenie for your configured Grafana integration.
|
|
|
|
|
Override priority | Configures the alert priority using the `og_priority` tag. The `og_priority` tag must have one of the following values: `P1`, `P2`, `P3`, `P4`, or `P5`. Default is `False`.
|
|
|
|
|
Send notification tags as | Specify how you would like [Notification Tags]({{< relref "create-alerts.md/#notifications" >}}) delivered to Opsgenie. They can be delivered as `Tags`, `Extra Properties` or both. Default is Tags. See note below for more information.
|
|
|
|
|
| Setting | Description |
|
|
|
|
|
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
|
|
|
| Alert API URL | The API URL for your Opsgenie instance. This will normally be either `https://api.opsgenie.com` or, for EU customers, `https://api.eu.opsgenie.com`. |
|
|
|
|
|
| API Key | The API Key as provided by Opsgenie for your configured Grafana integration. |
|
|
|
|
|
| Override priority | Configures the alert priority using the `og_priority` tag. The `og_priority` tag must have one of the following values: `P1`, `P2`, `P3`, `P4`, or `P5`. Default is `False`. |
|
|
|
|
|
| Send notification tags as | Specify how you would like [Notification Tags]({{< relref "create-alerts.md/#notifications" >}}) delivered to Opsgenie. They can be delivered as `Tags`, `Extra Properties` or both. Default is Tags. See note below for more information. |
|
|
|
|
|
|
|
|
|
|
> **Note:** When notification tags are sent as `Tags` they are concatenated into a string with a `key:value` format. If you prefer to receive the notifications tags as key/values under Extra Properties in Opsgenie then change the `Send notification tags as` to either `Extra Properties` or `Tags & Extra Properties`.
|
|
|
|
|
|
|
|
|
@@ -129,19 +129,19 @@ Send notification tags as | Specify how you would like [Notification Tags]({{< r
|
|
|
|
|
|
|
|
|
|
To set up PagerDuty, all you have to do is to provide an integration key.
|
|
|
|
|
|
|
|
|
|
Setting | Description
|
|
|
|
|
---------- | -----------
|
|
|
|
|
Integration Key | Integration key for PagerDuty.
|
|
|
|
|
Severity | Level for dynamic notifications, default is `critical` (1)
|
|
|
|
|
Auto resolve incidents | Resolve incidents in PagerDuty once the alert goes back to ok
|
|
|
|
|
Message in details | Removes the Alert message from the PD summary field and puts it into custom details instead (2)
|
|
|
|
|
| Setting | Description |
|
|
|
|
|
| ---------------------- | ----------------------------------------------------------------------------------------------- |
|
|
|
|
|
| Integration Key | Integration key for PagerDuty. |
|
|
|
|
|
| Severity | Level for dynamic notifications, default is `critical` (1) |
|
|
|
|
|
| Auto resolve incidents | Resolve incidents in PagerDuty once the alert goes back to ok |
|
|
|
|
|
| Message in details | Removes the Alert message from the PD summary field and puts it into custom details instead (2) |
|
|
|
|
|
|
|
|
|
|
>**Note:** The tags `Severity`, `Class`, `Group`, `dedup_key`, and `Component` have special meaning in the [Pagerduty Common Event Format - PD-CEF](https://support.pagerduty.com/docs/pd-cef). If an alert panel defines these tag keys, then they are transposed to the root of the event sent to Pagerduty. This means they will be available within the Pagerduty UI and Filtering tools. A Severity tag set on an alert overrides the global Severity set on the notification channel if it's a valid level.
|
|
|
|
|
> **Note:** The tags `Severity`, `Class`, `Group`, `dedup_key`, and `Component` have special meaning in the [Pagerduty Common Event Format - PD-CEF](https://support.pagerduty.com/docs/pd-cef). If an alert panel defines these tag keys, then they are transposed to the root of the event sent to Pagerduty. This means they will be available within the Pagerduty UI and Filtering tools. A Severity tag set on an alert overrides the global Severity set on the notification channel if it's a valid level.
|
|
|
|
|
|
|
|
|
|
>Using Message In Details will change the structure of the `custom_details` field in the PagerDuty Event.
|
|
|
|
|
This might break custom event rules in your PagerDuty rules if you rely on the fields in `payload.custom_details`.
|
|
|
|
|
Move any existing rules using `custom_details.myMetric` to `custom_details.queries.myMetric`.
|
|
|
|
|
This behavior will become the default in a future version of Grafana.
|
|
|
|
|
> Using Message In Details will change the structure of the `custom_details` field in the PagerDuty Event.
|
|
|
|
|
> This might break custom event rules in your PagerDuty rules if you rely on the fields in `payload.custom_details`.
|
|
|
|
|
> Move any existing rules using `custom_details.myMetric` to `custom_details.queries.myMetric`.
|
|
|
|
|
> This behavior will become the default in a future version of Grafana.
|
|
|
|
|
|
|
|
|
|
> **Note:** The `dedup_key` tag overrides the Grafana-generated `dedup_key` with a custom key.
|
|
|
|
|
|
|
|
|
@@ -152,22 +152,22 @@ This behavior will become the default in a future version of Grafana.
|
|
|
|
|
To configure VictorOps, provide the URL from the Grafana Integration and substitute `$routing_key` with a valid key.
|
|
|
|
|
|
|
|
|
|
> **Note:** The tag `Severity` has special meaning in the [VictorOps Incident Fields](https://help.victorops.com/knowledge-base/incident-fields-glossary/). If an alert panel defines this key, then it replaces the `message_type` in the root of the event sent to VictorOps.
|
|
|
|
|
|
|
|
|
|
### Pushover
|
|
|
|
|
|
|
|
|
|
To set up Pushover, you must provide a user key and an API token. Refer to [What is Pushover and how do I use it](https://support.pushover.net/i7-what-is-pushover-and-how-do-i-use-it) for instructions on how to generate them.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Setting | Description
|
|
|
|
|
---------- | -----------
|
|
|
|
|
API Token | Application token
|
|
|
|
|
User key(s) | A comma-separated list of user keys
|
|
|
|
|
Device(s) | A comma-separated list of devices
|
|
|
|
|
Priority | The priority alerting nottifications are sent
|
|
|
|
|
OK priority | The priority OK notifications are sent; if not set, then OK notifications are sent with the priority set for alerting notifications
|
|
|
|
|
Retry | How often (in seconds) the Pushover servers send the same notification to the user. (minimum 30 seconds)
|
|
|
|
|
Expire | How many seconds your notification will continue to be retried for (maximum 86400 seconds)
|
|
|
|
|
Alerting sound | The sound for alerting notifications
|
|
|
|
|
OK sound | The sound for OK notifications
|
|
|
|
|
| Setting | Description |
|
|
|
|
|
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
|
|
|
|
|
| API Token | Application token |
|
|
|
|
|
| User key(s) | A comma-separated list of user keys |
|
|
|
|
|
| Device(s) | A comma-separated list of devices |
|
|
|
|
|
| Priority | The priority alerting nottifications are sent |
|
|
|
|
|
| OK priority | The priority OK notifications are sent; if not set, then OK notifications are sent with the priority set for alerting notifications |
|
|
|
|
|
| Retry | How often (in seconds) the Pushover servers send the same notification to the user. (minimum 30 seconds) |
|
|
|
|
|
| Expire | How many seconds your notification will continue to be retried for (maximum 86400 seconds) |
|
|
|
|
|
| Alerting sound | The sound for alerting notifications |
|
|
|
|
|
| OK sound | The sound for OK notifications |
|
|
|
|
|
|
|
|
|
|
### Webhook
|
|
|
|
|
|
|
|
|
@@ -178,26 +178,26 @@ Example json body:
|
|
|
|
|
|
|
|
|
|
```json
|
|
|
|
|
{
|
|
|
|
|
"dashboardId":1,
|
|
|
|
|
"evalMatches":[
|
|
|
|
|
"dashboardId": 1,
|
|
|
|
|
"evalMatches": [
|
|
|
|
|
{
|
|
|
|
|
"value":1,
|
|
|
|
|
"metric":"Count",
|
|
|
|
|
"tags":{}
|
|
|
|
|
"value": 1,
|
|
|
|
|
"metric": "Count",
|
|
|
|
|
"tags": {}
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"imageUrl":"https://grafana.com/assets/img/blog/mixed_styles.png",
|
|
|
|
|
"message":"Notification Message",
|
|
|
|
|
"orgId":1,
|
|
|
|
|
"panelId":2,
|
|
|
|
|
"ruleId":1,
|
|
|
|
|
"ruleName":"Panel Title alert",
|
|
|
|
|
"ruleUrl":"http://localhost:3000/d/hZ7BuVbWz/test-dashboard?fullscreen\u0026edit\u0026tab=alert\u0026panelId=2\u0026orgId=1",
|
|
|
|
|
"state":"alerting",
|
|
|
|
|
"tags":{
|
|
|
|
|
"tag name":"tag value"
|
|
|
|
|
"imageUrl": "https://grafana.com/assets/img/blog/mixed_styles.png",
|
|
|
|
|
"message": "Notification Message",
|
|
|
|
|
"orgId": 1,
|
|
|
|
|
"panelId": 2,
|
|
|
|
|
"ruleId": 1,
|
|
|
|
|
"ruleName": "Panel Title alert",
|
|
|
|
|
"ruleUrl": "http://localhost:3000/d/hZ7BuVbWz/test-dashboard?fullscreen\u0026edit\u0026tab=alert\u0026panelId=2\u0026orgId=1",
|
|
|
|
|
"state": "alerting",
|
|
|
|
|
"tags": {
|
|
|
|
|
"tag name": "tag value"
|
|
|
|
|
},
|
|
|
|
|
"title":"[Alerting] Panel Title alert"
|
|
|
|
|
"title": "[Alerting] Panel Title alert"
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
@@ -213,7 +213,7 @@ In DingTalk PC Client:
|
|
|
|
|
|
|
|
|
|
2. Click "Robot Manage" item in the pop menu, there will be a new panel call "Robot Manage".
|
|
|
|
|
|
|
|
|
|
3. In the "Robot Manage" panel, select "customized: customized robot with Webhook".
|
|
|
|
|
3. In the "Robot Manage" panel, select "customized: customized robot with Webhook".
|
|
|
|
|
|
|
|
|
|
4. In the next new panel named "robot detail", click "Add" button.
|
|
|
|
|
|
|
|
|
@@ -226,11 +226,11 @@ In DingTalk PC Client:
|
|
|
|
|
To set up Discord, you must create a Discord channel webhook. For instructions on how to create the channel, refer to
|
|
|
|
|
[Intro to Webhooks](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks).
|
|
|
|
|
|
|
|
|
|
Setting | Description
|
|
|
|
|
---------- | -----------
|
|
|
|
|
Webhook URL | Discord webhook URL.
|
|
|
|
|
Message Content | Mention a group using @ or a user using <@ID> when notifying in a channel.
|
|
|
|
|
Avatar URL | Optionally, provide a URL to an image to use as the avatar for the bot's message.
|
|
|
|
|
| Setting | Description |
|
|
|
|
|
| --------------- | --------------------------------------------------------------------------------- |
|
|
|
|
|
| Webhook URL | Discord webhook URL. |
|
|
|
|
|
| Message Content | Mention a group using @ or a user using <@ID> when notifying in a channel. |
|
|
|
|
|
| Avatar URL | Optionally, provide a URL to an image to use as the avatar for the bot's message. |
|
|
|
|
|
|
|
|
|
|
Alternately, use the [Slack](#slack) notifier by appending `/slack` to a Discord webhook URL.
|
|
|
|
|
|
|
|
|
|