public/grafana
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Revert "Alerting docs: rework create alert rules definition and topic" (#83328)

Browse Source
This commit is contained in:
Jack Baldry
2024-02-23 19:30:08 +02:00
committed by GitHub
parent 18d1ced069
commit 2b4f108771
21 changed files with 188 additions and 171 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sources/alerting/configure-notifications/_index.md
-26
View File
@@ -1,26 +0,0 @@
---
canonical: https://grafana.com/docs/grafana/latest/alerting/configure-notifications
description: Configure how, when, and where to send your alert notifications
keywords:
- grafana
- alert
- notifications
labels:
products:
- cloud
- enterprise
- oss
menuTitle: Configure notifications
title: Configure notifications
weight: 125
---
# Configure notifications
Choose how, when, and where to send your alert notifications.
As a first step, define your contact points; where to send your alert notifications to. A contact point is a set of one or more integrations that are used to deliver notifications.
Next, create a notification policy which is a set of rules for where, when and how your alerts are routed to contact points. In a notification policy, you define where to send your alert notifications by choosing one of the contact points you created.
Optionally, you can add notification templates to contact points for reuse and consistent messaging in your notifications.
docs/sources/alerting/configure-notifications/create-notification-policy.md
-117
View File
@@ -1,117 +0,0 @@
---
aliases:
- ../notifications/
- ../old-alerting/notifications/
- ../unified-alerting/notifications/
- ./alerting-rules/
canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-notification-policy/
description: Configure notification policies to determine how alerts are routed to contact points
keywords:
- grafana
- alerting
- guide
- notification policies
- routes
labels:
products:
- cloud
- enterprise
- oss
title: Configure notification policies
weight: 430
---
# Configure notification policies
Notification policies determine how alerts are routed to contact points.
Policies have a tree structure, where each policy can have one or more nested policies. Each policy, except for the default policy, can also match specific alert labels.
Each alert is evaluated by the default policy and subsequently by each nested policy.
If the **Continue matching subsequent sibling nodes** option is enabled for a nested policy, then evaluation continues even after one or more matches. A parent policy’s configuration settings and contact point information govern the behavior of an alert that does not match any of the nested policies. A default policy governs any alert that does not match a nested policy.
You can configure Grafana-managed notification policies as well as notification policies for an external Alertmanager data source.
For more information on notification policies, see [fundamentals of Notification Policies][notification-policies].
## Edit default notification policy
{{% admonition type="note" %}}
Before Grafana v8.2, the configuration of the embedded Alertmanager was shared across organizations. Users of Grafana 8.0 and 8.1 are advised to use the new Grafana 8 Alerts only if they have one organization. Otherwise, silences for the Grafana managed alerts will be visible by all organizations.
{{% /admonition %}}
1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Notification policies**.
1. From the **Choose Alertmanager** dropdown, select an external Alertmanager. By default, the **Grafana Alertmanager** is selected.
1. In the Default policy section, click **...** -> **Edit**.
1. In **Default contact point**, update the contact point to whom notifications should be sent for rules when alert rules do not match any specific policy.
1. In **Group by**, choose labels to group alerts by. If multiple alerts are matched for this policy, then they are grouped by these labels. A notification is sent per group. If the field is empty (default), then all notifications are sent in a single group. Use a special label `...` to group alerts by all labels (which effectively disables grouping).
1. In **Timing options**, select from the following options:
- **Group wait** Time to wait to buffer alerts of the same group before sending an initial notification. Default is 30 seconds.
- **Group interval** Minimum time interval between two notifications for a group. Default is 5 minutes.
- **Repeat interval** Minimum time interval for re-sending a notification if no new alerts were added to the group. Default is 4 hours.
1. Click **Save** to save your changes.
## Add new nested policy
To create a new notification policy, you need to follow its tree structure. New policies created on the trunk of the tree (default policy), are the tree branches. And, subsequently, each branch can bear their own child policies. This is why you will always be adding a new **nested** policy under either the default policy, or under a already nested policy.
1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Notification policies**.
1. From the **Choose Alertmanager** dropdown, select an Alertmanager. By default, the **Grafana Alertmanager** is selected.
1. To add a top level specific policy, go to the Specific routing section (either to the default policy, or to another existing policy in which you would like to add a new nested policy) and click **+New nested policy**.
1. In the Matching labels section, add one or more rules for matching alert labels.
1. In the **Contact point** dropdown, select the contact point to send notification to if alert matches only this specific policy and not any of the nested policies.
1. Optionally, enable **Continue matching subsequent sibling nodes** to continue matching sibling policies even after the alert matched the current policy. When this option is enabled, you can get more than one notification for one alert.
1. Optionally, enable **Override grouping** to specify the same grouping as the default policy. If this option is not enabled, the default policy grouping is used.
1. Optionally, enable **Override general timings** to override the timing options configured in the group notification policy.
1. Click **Save policy** to save your changes.
## Add nested policy
1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Notification policies**.
1. Expand the specific policy you want to update.
1. Click **+ Add nested policy**, then add the details using information in [Add new specific policy](#add-new-nested-policy).
1. Click **Save policy** to save your changes.
## Edit notification policies
1. In the left-side menu, click **Alerts & IRM**, and then **Alerting**.
1. Click **Notification policies**.
1. Find the policy you want to edit, then click **...** -> **Edit**.
1. Make any changes using instructions in [Add new specific policy](#add-new-nested-policy).
1. Save your changes.
## Searching for policies
Grafana allows you to search within the tree of policies by the following:
- **Label matchers**
- **Contact Points**
To search by contact point simply select a contact point from the **Search by contact point** dropdown. The policies that use that contact point will be highlighted in the user interface.
To search by label matchers simply enter a valid matcher in the **Search by matchers** input field. Multiple matchers can be combined with a comma (`,`).
An example of a valid matchers search input is:
`severity=high, region=~EMEA|NASA`
> All matched policies will be **exact** matches, we currently do not support regex-style or partial matching.
## Example
An example of an alert configuration.
- Create a "default" contact point for slack notifications, and set it on the default policy.
- Edit the default policy grouping to group alerts by `cluster`, `namespace` and `severity` so that you get a notification per alert rule and specific kubernetes cluster and namespace.
- Create specific route for alerts coming from the development cluster with an appropriate contact point.
- Create a specific route for alerts with "critical" severity with a more invasive contact point integration, like pager duty notification.
- Create specific routes for particular teams that handle their own on-call rotations.
{{% docs/reference %}}
[notification-policies]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notification-policies/notifications"
[notification-policies]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notification-policies/notifications"
{{% /docs/reference %}}
docs/sources/alerting/configure-notifications/create-silence.md
-75
View File
@@ -1,75 +0,0 @@
---
aliases:
- ../silences/create-silence/
- ../silences/edit-silence/
- ../silences/linking-to-silence-form/
- ../silences/remove-silence/
- ../unified-alerting/silences/
- ../silences/
- ./alerting/manage-notifications/create-silence/
canonical: https://grafana.com/docs/grafana/latest/alerting/manage-notifications/create-silence/
description: Create silences to stop notifications from getting created for a specified window of time
keywords:
- grafana
- alerting
- silence
- mute
labels:
products:
- cloud
- enterprise
- oss
title: Manage silences
weight: 440
---
# Manage silences
Silences stop notifications from getting created and last for only a specified window of time.
**Note that inhibition rules are not supported in the Grafana Alertmanager.**
## Add silences
To add a silence, complete the following steps.
1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Silences**.
1. From the **Alertmanager** dropdown, select an external Alertmanager to create and manage silences for the external data source. Otherwise, keep the default option of Grafana.
1. Click **Create silence** to open the Create silence page.
1. In **Silence start and end**, select the start and end date to indicate when the silence should go into effect and expire.
1. Optionally, in **Duration**, specify how long the silence is enforced. This automatically updates the end time in the **Silence start and end** field.
1. In the **Label** and **Value** fields, enter one or more _Matching Labels_. Matchers determine which rules the silence will apply to.
1. In **Comment**, add details about the silence.
1. Click **Submit**.
## Edit silences
To edit a silence, complete the following steps.
1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Silences** to view the list of existing silences.
1. Find the silence you want to edit, then click **Edit** (pen icon).
1. Make the desired changes, then click **Submit** to save your changes.
## Create a URL to link to a silence form
When linking to a silence form, provide the default matching labels and comment via `matcher` and `comment` query parameters. The `matcher` parameter should be in the following format `[label][operator][value]` where the `operator` parameter can be one of the following: `=` (equals, not regex), `!=` (not equals, not regex), `=~` (equals, regex), `!~` (not equals, regex).
The URL can contain many query parameters with the key `matcher`.
For example, to link to silence form with matching labels `severity=critical` & `cluster!~europe-.*` and comment `Silence critical EU alerts`, create a URL `https://mygrafana/alerting/silence/new?matcher=severity%3Dcritical&matcher=cluster!~europe-*&comment=Silence%20critical%20EU%20alert`.
To link to a new silence page for an external Alertmanager, add a `alertmanager` query parameter with the Alertmanager data source name.
## Remove silences
To remove a silence, complete the following steps.
1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Silences** to view the list of existing silences.
1. Select the silence you want to end, then click **Unsilence**.
> **Note:** You cannot remove a silence manually. Silences that have ended are retained and listed for five days.
## Useful links
[Aggregation operators](https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators)
docs/sources/alerting/configure-notifications/manage-contact-points/_index.md
-142
View File
@@ -1,142 +0,0 @@
---
aliases:
- ../contact-points/ # /docs/grafana/<GRAFANA_VERSION>/alerting/contact-points/
- ../contact-points/create-contact-point/ # /docs/grafana/<GRAFANA_VERSION>/alerting/contact-points/create-contact-point/
- ../contact-points/delete-contact-point/ # /docs/grafana/<GRAFANA_VERSION>/alerting/contact-points/delete-contact-point/
- ../contact-points/edit-contact-point/ # /docs/grafana/<GRAFANA_VERSION>/alerting/contact-points/edit-contact-point/
- ../contact-points/test-contact-point/ # /docs/grafana/<GRAFANA_VERSION>/alerting/contact-points/test-contact-point/
- ../manage-notifications/manage-contact-points/ # /docs/grafana/<GRAFANA_VERSION>/alerting/manage-notifications/manage-contact-points/
- create-contact-point/ # /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-contact-point/
- ./alerting-rules/
- ./alerting-rules/manage-notifications/manage-contact-points/
- alerting/manage-notifications/manage-contact-points/configure-integrations/
- ./alerting-rules/manage-contact-points/
canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/manage-contact-points/
description: Configure contact points to define how your contacts are notified when an alert rule fires
keywords:
- grafana
- alerting
- guide
- contact point
- templating
labels:
products:
- cloud
- enterprise
- oss
title: Configure contact points
weight: 410
---
# Configure contact points
Use contact points to define how your contacts are notified when an alert rule fires. You can add, edit, delete, and test a contact point.
Configure contact point integrations in Grafana to select your preferred communication channel for receiving notifications when your alert rules are firing.
## Add a contact point
Complete the following steps to add a contact point.
1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Contact points**.
1. From the **Choose Alertmanager** dropdown, select an Alertmanager. By default, **Grafana Alertmanager** is selected.
1. On the **Contact Points** tab, click **+ Add contact point**.
1. Enter a descriptive name for the contact point.
1. From **Integration**, select a type and fill out mandatory fields. For example, if you choose email, enter the email addresses. Or if you choose Slack, enter the Slack channel(s) and users who should be contacted.
1. Some contact point integrations, like email or webhook, have optional settings. In **Optional settings**, specify additional settings for the selected contact point integration.
1. In Notification settings, optionally select **Disable resolved message** if you do not want to be notified when an alert resolves.
1. To add another contact point integration, click **Add contact point integration** and repeat steps 6 through 8.
1. Save your changes.
## Edit a contact point
Complete the following steps to edit a contact point.
1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Contact points** to view a list of existing contact points.
1. On the **Contact Points** tab, find the contact point you want to edit, and then click **Edit**.
1. Update the contact point and save your changes.
## Delete a contact point
Complete the following steps to delete a contact point.
1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Contact points** to view a list of existing contact points.
1. On the **Contact Points** tab, find the contact point you want to delete, and then click **More** -> **Delete**.
1. In the confirmation dialog, click **Yes, delete**.
{{% admonition type="note" %}}
You cannot delete contact points that are in use by a notification policy. Either delete the notification policy or update it to use another contact point.
{{% /admonition %}}
## Test a contact point
Complete the following steps to test a contact point.
1. In the left-side menu, click **Alerts & IRM** and then **Alerting**.
1. Click **Contact points** to view a list of existing contact points.
1. On the **Contact Points** tab, find the contact point you want to test, then click **Edit**. You can also create a new contact point if needed.
1. Click **Test** to open the contact point testing modal.
1. Choose whether to send a predefined test notification or choose custom to add your own custom annotations and labels to include in the notification.
1. Click **Send test notification** to fire the alert.
## Manage contact points
The Contact points list view lists all existing contact points and notification templates.
On the **Contact Points** tab, you can:
- Search for name and type of contact points and integrations
- View all existing contact points and integrations
- View how many notification policies each contact point is being used for and navigate directly to the linked notification policies
- View the status of notification deliveries
- Export individual contact points or all contact points in JSON, YAML, or Terraform format
- Delete contact points that are not in use by a notification policy
On the **Notification templates** tab, you can:
- View, edit, copy or delete existing notification templates
## Configure contact point integrations
Each contact point integration has its own configuration options and setup process. In most cases, this involves providing an API key or a Webhook URL.
Once configured, you can use integrations as part of your contact points to receive notifications whenever your alert changes its state. In this section, we'll cover the basic steps to configure your integrations, so you can start receiving real-time alerts and stay on top of your monitoring data.
## List of supported integrations
| Name | Type |
| ------------------------ | ------------------------- |
| DingDing | `dingding` |
| Discord | `discord` |
| Email | `email` |
| Google Chat | `googlechat` |
| [Grafana Oncall][oncall] | `oncall` |
| Hipchat | `hipchat` |
| Kafka | `kafka` |
| Line | `line` |
| Microsoft Teams | `teams` |
| Opsgenie | `opsgenie` |
| [Pagerduty][pagerduty] | `pagerduty` |
| Prometheus Alertmanager | `prometheus-alertmanager` |
| Pushover | `pushover` |
| Sensu | `sensu` |
| Sensu Go | `sensugo` |
| Slack | `slack` |
| Telegram | `telegram` |
| Threema | `threema` |
| VictorOps | `victorops` |
| [Webhook][webhook] | `webhook` |
{{% docs/reference %}}
[pagerduty]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/integrations/pager-duty"
[pagerduty]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/manage-contact-points/integrations/pager-duty"
[oncall]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/integrations/configure-oncall"
[oncall]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/manage-contact-points/integrations/configure-oncall"
[webhook]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier"
[webhook]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier"
{{% /docs/reference %}}
docs/sources/alerting/configure-notifications/manage-contact-points/integrations/configure-oncall.md
-74
View File
@@ -1,74 +0,0 @@
---
canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/manage-contact-points/integrations/configure-oncall/
description: Configure the Alerting - Grafana OnCall integration to connect alerts generated by Grafana Alerting with Grafana OnCall
keywords:
- grafana
- alerting
- oncall
- integration
aliases:
- ../configure-oncall/ # /docs/grafana/latest/alerting/alerting-rules/manage-contact-points/configure-oncall/
labels:
products:
- cloud
- enterprise
- oss
menuTitle: Grafana OnCall
title: Configure Grafana OnCall for Alerting
weight: 300
---
## Configure Grafana OnCall for Alerting
Use the Grafana Alerting - Grafana OnCall integration to effortlessly connect alerts generated by Grafana Alerting with Grafana OnCall, where you can then route them according to defined escalation chains and schedules.
You can set up the integration using the Grafana Alerting application or the Grafana OnCall application. For more information on setting it up from the Grafana OnCall application, see [Grafana OnCall documentation][oncall-integration].
### Before you begin
- Ensure you have Installed and enabled the Grafana OnCall plugin
- Ensure your version of Grafana is up-to-date and supports the new features
### Procedure
To set up the Grafana OnCall integration using the Grafana Alerting application, complete the following steps.
1. Navigate to **Alerts&IRM** -> **Alerting** -> **Contact points**.
1. Click **+ Add contact point**.
1. Enter a contact point name.
1. From the Integration list, select Grafana OnCall.
**Note:**
The Grafana OnCall integration is only available for Grafana Alertmanager.
1. Choose whether to add a new OnCall integration or add an existing one.
- If you add a new one, enter an Integration name.
- If you add an existing one, choose from the list of available integrations
1. Click **Save contact point**.
1. On the Contact points list view page, you can see the contact point with the Grafana OnCall icon.
If the integration is not yet being used anywhere in the notification policies tree, it will have **Unused** as its status in the **Health** column. It won’t receive any notifications, because there are no notifications using that integration.
1. Connect your contact point to a notification policy.
For more information on connecting your contact point to a notification policy, see
[Create notification policy][create-notification-policy].
1. To view your integration in the Grafana OnCall application and set up routes and escalation chains, click the Link next to the integration on the Contact points list view page in the **Type** column.
This redirects you to the Grafana OnCall integration page in the Grafana OnCall application. From there, you can add [routes and escalation chains][escalation-chain].
{{% docs/reference %}}
[create-notification-policy]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-notification-policy"
[create-notification-policy]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/create-notification-policy"
[oncall-integration]: "/docs/grafana/ -> /docs/oncall/latest/integrations/grafana-alerting"
[oncall-integration]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/integrations/grafana-alerting"
[escalation-chain]: "/docs/grafana/ -> /docs/oncall/latest/configure/escalation-chains-and-routes"
[escalation-chain]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/configure/escalation-chains-and-routes"
{{% /docs/reference %}}
docs/sources/alerting/configure-notifications/manage-contact-points/integrations/pager-duty.md
-41
View File
@@ -1,41 +0,0 @@
---
canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/manage-contact-points/integrations/pager-duty/
description: Configure the PagerDuty integration for Alerting
keywords:
- grafana
- alerting
- pagerduty
labels:
products:
- cloud
- enterprise
- oss
menuTitle: PagerDuty
title: Configure PagerDuty for Alerting
weight: 400
---
# Configure PagerDuty for Alerting
To set up PagerDuty, provide an integration key.
| Setting | Description |
| --------------- | ------------------------------------------------------ |
| Integration Key | Integration key for PagerDuty |
| Severity | Level for dynamic notifications, default is `critical` |
| Custom Details | Additional details about the event |
The `CustomDetails` field is an object containing arbitrary key-value pairs. The user-defined details are merged with the ones we use by default.
Our default values for `CustomDetails` are:
```go
{
"firing": `{{ template "__text_alert_list" .Alerts.Firing }}`,
"resolved": `{{ template "__text_alert_list" .Alerts.Resolved }}`,
"num_firing": `{{ .Alerts.Firing | len }}`,
"num_resolved": `{{ .Alerts.Resolved | len }}`,
}
```
In case of duplicate keys, the user-defined details overwrite the default ones.
docs/sources/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier.md
-150
View File
@@ -1,150 +0,0 @@
---
aliases:
- ../contact-points/notifiers/webhook-notifier/
- ../fundamentals/contact-points/webhook-notifier/
- alerting/manage-notifications/manage-contact-points/webhook-notifier/
canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/manage-contact-points/integrations/webhook-notifier/
description: Configure the webhook notifier integration for Alerting
keywords:
- grafana
- alerting
- guide
- contact point
- templating
labels:
products:
- cloud
- enterprise
- oss
menuTitle: Webhook notifier
title: Configure the webhook notifier for Alerting
weight: 200
---
### Configure the webhook notifier for Alerting
Example JSON body:
```json
{
"receiver": "My Super Webhook",
"status": "firing",
"orgId": 1,
"alerts": [
{
"status": "firing",
"labels": {
"alertname": "High memory usage",
"team": "blue",
"zone": "us-1"
},
"annotations": {
"description": "The system has high memory usage",
"runbook_url": "https://myrunbook.com/runbook/1234",
"summary": "This alert was triggered for zone us-1"
},
"startsAt": "2021-10-12T09:51:03.157076+02:00",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "https://play.grafana.org/alerting/1afz29v7z/edit",
"fingerprint": "c6eadffa33fcdf37",
"silenceURL": "https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT2%2Cteam%3Dblue%2Czone%3Dus-1",
"dashboardURL": "",
"panelURL": "",
"values": {
"B": 44.23943737541908,
"C": 1
}
},
{
"status": "firing",
"labels": {
"alertname": "High CPU usage",
"team": "blue",
"zone": "eu-1"
},
"annotations": {
"description": "The system has high CPU usage",
"runbook_url": "https://myrunbook.com/runbook/1234",
"summary": "This alert was triggered for zone eu-1"
},
"startsAt": "2021-10-12T09:56:03.157076+02:00",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "https://play.grafana.org/alerting/d1rdpdv7k/edit",
"fingerprint": "bc97ff14869b13e3",
"silenceURL": "https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT1%2Cteam%3Dblue%2Czone%3Deu-1",
"dashboardURL": "",
"panelURL": "",
"values": {
"B": 44.23943737541908,
"C": 1
}
}
],
"groupLabels": {},
"commonLabels": {
"team": "blue"
},
"commonAnnotations": {},
"externalURL": "https://play.grafana.org/",
"version": "1",
"groupKey": "{}:{}",
"truncatedAlerts": 0,
"title": "[FIRING:2] (blue)",
"state": "alerting",
"message": "**Firing**\n\nLabels:\n - alertname = T2\n - team = blue\n - zone = us-1\nAnnotations:\n - description = This is the alert rule checking the second system\n - runbook_url = https://myrunbook.com\n - summary = This is my summary\nSource: https://play.grafana.org/alerting/1afz29v7z/edit\nSilence: https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT2%2Cteam%3Dblue%2Czone%3Dus-1\n\nLabels:\n - alertname = T1\n - team = blue\n - zone = eu-1\nAnnotations:\nSource: https://play.grafana.org/alerting/d1rdpdv7k/edit\nSilence: https://play.grafana.org/alerting/silence/new?alertmanager=grafana&matchers=alertname%3DT1%2Cteam%3Dblue%2Czone%3Deu-1\n"
}
```
### Webhook fields
## Body
| Key | Type | Description |
| ----------------- | ------------------------- | ------------------------------------------------------------------------------- |
| receiver | string | Name of the webhook |
| status | string | Current status of the alert, `firing` or `resolved` |
| orgId | number | ID of the organization related to the payload |
| alerts | array of [alerts](#alert) | Alerts that are triggering |
| groupLabels | object | Labels that are used for grouping, map of string keys to string values |
| commonLabels | object | Labels that all alarms have in common, map of string keys to string values |
| commonAnnotations | object | Annotations that all alarms have in common, map of string keys to string values |
| externalURL | string | External URL to the Grafana instance sending this webhook |
| version | string | Version of the payload |
| groupKey | string | Key that is used for grouping |
| truncatedAlerts | number | Number of alerts that were truncated |
| title | string | **Will be deprecated soon** |
| state | string | **Will be deprecated soon** |
| message | string | **Will be deprecated soon** |
### Alert
| Key | Type | Description |
| ------------ | ------ | ---------------------------------------------------------------------------------- |
| status | string | Current status of the alert, `firing` or `resolved` |
| labels | object | Labels that are part of this alert, map of string keys to string values |
| annotations | object | Annotations that are part of this alert, map of string keys to string values |
| startsAt | string | Start time of the alert |
| endsAt | string | End time of the alert, default value when not resolved is `0001-01-01T00:00:00Z` |
| values | object | Values that triggered the current status |
| generatorURL | string | URL of the alert rule in the Grafana UI |
| fingerprint | string | The labels fingerprint, alarms with the same labels will have the same fingerprint |
| silenceURL | string | URL to silence the alert rule in the Grafana UI |
| dashboardURL | string | **Will be deprecated soon** |
| panelURL | string | **Will be deprecated soon** |
| imageURL | string | URL of a screenshot of a panel assigned to the rule that created this notification |
### Removed fields related to dashboards
Alerts are not coupled to dashboards anymore therefore the fields related to dashboards `dashboardId` and `panelId` have been removed.
## WeCom
WeCom contact points need a Webhook URL. These are obtained by setting up a WeCom robot on the corresponding group chat. To obtain a Webhook URL using the WeCom desktop Client please follow these steps:
1. Click the "..." in the top right corner of a group chat that you want your alerts to be delivered to
2. Click "Add Group Robot", select "New Robot" and give your robot a name. Click "Add Robot"
3. There should be a Webhook URL in the panel.
| Setting | Description |
| ------- | ------------------ |
| Url | WeCom webhook URL. |
docs/sources/alerting/configure-notifications/mute-timings.md
-90
View File
@@ -1,90 +0,0 @@
---
aliases:
- ../notifications/mute-timings/
- ../unified-alerting/notifications/mute-timings/
- ./alerting/manage-notifications/mute-timings/
canonical: https://grafana.com/docs/grafana/latest/alerting/manage-notifications/mute-timings/
description: Create mute timings to prevent alerts from firing during a specific and reoccurring period of time
keywords:
- grafana
- alerting
- guide
- mute
- mute timings
- mute time interval
labels:
products:
- cloud
- enterprise
- oss
title: Create mute timings
weight: 450
---
# Create mute timings
A mute timing is a recurring interval of time when no new notifications for a policy are generated or sent. Use them to prevent alerts from firing a specific and reoccurring period, for example, a regular maintenance period.
Similar to silences, mute timings do not prevent alert rules from being evaluated, nor do they stop alert instances from being shown in the user interface. They only prevent notifications from being created.
You can configure Grafana managed mute timings as well as mute timings for an [external Alertmanager data source][datasources/alertmanager]. For more information, refer to [Alertmanager documentation][fundamentals/alertmanager].
## Mute timings vs silences
The following table highlights the key differences between mute timings and silences.
| Mute timing | Silence |
| -------------------------------------------------- | ---------------------------------------------------------------------------- |
| Uses time interval definitions that can reoccur | Has a fixed start and end time |
| Is created and then added to notification policies | Uses labels to match against an alert to determine whether to silence or not |
## Add mute timings
1. In the left-side menu, click **Alerts & IRM**, and then **Alerting**.
1. Click **Notification policies** and then the **Mute Timings** tab.
1. From the **Alertmanager** dropdown, select an external Alertmanager. By default, the **Grafana Alertmanager** is selected.
1. Click **+ Add mute timing**.
1. Fill out the form to create a [time interval](#time-intervals) to match against for your mute timing.
1. Save your mute timing.
## Add mute timing to a notification policy
1. In the left-side menu, click **Alerts & IRM**, and then **Alerting**.
1. Click **Notification policies** and make sure you are on the **Notification Policies** tab.
1. Find the notification policy you would like to add the mute timing to and click **...** -> **Edit**.
1. From the **Mute timings** dropdown, choose the mute timings you would like to add to the policy.
1. Save your changes.
## Time intervals
A time interval is a specific duration during which alerts are suppressed. The duration typically consists of a specific time range and the days of the week, month, or year.
Supported time interval options are:
- Time range: The time inclusive of the start and exclusive of the end time (in UTC if no location has been selected, otherwise local time).
- Location: Depending on the location you select, the time range is displayed in local time.
- Days of the week: The day or range of days of the week. Example: `monday:thursday`.
- Days of the month: The date 1-31 of a month. Negative values can also be used to represent days that begin at the end of the month. For example: `-1` for the last day of the month.
- Months: The months of the year in either numerical or the full calendar month. For example: `1, may:august`.
- Years: The year or years for the interval. For example: `2021:2024`.
All fields are lists; to match the field, at least one list element must be satisfied. Fields also support ranges using `:` (e.g., `monday:thursday`).
If a field is left blank, any moment of time will match the field. For an instant of time to match a complete time interval, all fields must match. A mute timing can contain multiple time intervals.
If you want to specify an exact duration, specify all the options. For example, if you wanted to create a time interval for the first Monday of the month, for March, June, September, and December, between the hours of 12:00 and 24:00 UTC your time interval specification would be:
- Time range:
- Start time: `12:00`
- End time: `24:00`
- Days of the week: `monday`
- Months: `3, 6, 9, 12`
- Days of the month: `1:7`
{{% docs/reference %}}
[datasources/alertmanager]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/datasources/alertmanager"
[datasources/alertmanager]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/connect-externally-hosted/data-sources/alertmanager"
[fundamentals/alertmanager]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alertmanager"
[fundamentals/alertmanager]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alertmanager"
{{% /docs/reference %}}
docs/sources/alerting/configure-notifications/template-notifications/_index.md
-65
View File
@@ -1,65 +0,0 @@
---
aliases:
- ./alerting-rules/manage-notifications/manage-contact-points/template-notifications/
canonical: https://grafana.com/docs/grafana/latest/alerting/manage-notifications/template-notifications/
description: Customize your notifications using notification templates
keywords:
- grafana
- alerting
- notifications
- templates
labels:
products:
- cloud
- enterprise
- oss
title: Customize notifications
weight: 420
---
# Customize notifications
Customize your notifications with notifications templates.
You can use notification templates to change the title, message, and format of the message in your notifications.
Notification templates are not tied to specific contact point integrations, such as email or Slack. However, you can choose to create separate notification templates for different contact point integrations.
You can use notification templates to:
- Customize the subject of an email or the title of a message.
- Add, change or remove text in notifications. For example, to select or omit certain labels, annotations and links.
- Format text in bold and italic, and add or remove line breaks.
You cannot use notification templates to:
- Add HTML and CSS to email notifications to change their visual appearance.
- Change the design of notifications in instant messaging services such as Slack and Microsoft Teams. For example, to add or remove custom blocks with Slack Block Kit or adaptive cards with Microsoft Teams.
- Choose the number and size of images, or where in the notification images are shown.
- Customize the data in webhooks, including the fields or structure of the JSON data or send the data in other formats such as XML.
- Add or remove HTTP headers in webhooks other than those in the contact point configuration.
[Using Go's templating language][using-go-templating-language]
Learn how to write the content of your notification templates in Go’s templating language.
Create reusable notification templates for your contact points.
[Use notification templates][use-notification-templates]
Use notification templates to send notifications to your contact points.
[Reference][reference]
Data that is available when writing templates.
{{% docs/reference %}}
[reference]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference"
[reference]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference"
[use-notification-templates]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/use-notification-templates"
[use-notification-templates]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/use-notification-templates"
[using-go-templating-language]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/using-go-templating-language"
[using-go-templating-language]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/using-go-templating-language"
{{% /docs/reference %}}
docs/sources/alerting/configure-notifications/template-notifications/create-notification-templates.md
-296
View File
@@ -1,296 +0,0 @@
---
aliases:
- ./alerting-rules/manage-notifications/manage-contact-points/template-notifications/create-notification-templates/
canonical: https://grafana.com/docs/grafana/latest/alerting/manage-notifications/template-notifications/create-notification-templates/
description: Create notification templates to sent to your contact points
keywords:
- grafana
- alerting
- notifications
- templates
- create templates
- edit templates
- delete templates
labels:
products:
- cloud
- enterprise
- oss
title: Create notification templates
weight: 200
---
# Create notification templates
Create reusable notification templates to send to your contact points.
You can add one or more templates to your notification template.
Your notification template name must be unique. You cannot have two templates with the same name in the same notification template or in different notification templates. Avoid defining templates with the same name as default templates, such as: `__subject`, `__text_values_list`, `__text_alert_list`, `default.title` and `default.message`.
To create a notification template, complete the following steps.
1. Click **Alerts & IRM** -> **Contact points**.
1. Click the **Notification Templates** tab and then **+ Add notification template**.
1. Enter a name for the notification template.
1. Write the content of the template in the content field.
1. Save your changes.
`{{ define "email.subject" }}` and `{{ end }}` is automatically added to the start and end of the content:
To create a notification template that contains more than one template:
1. Click **+ Add notification template**.
2. Enter a name for the notification template.
3. Write each template in the Content field, including `{{ define "name-of-template" }}` and `{{ end }}` at the start and end of each template.
4. Save your changes.
## Preview notification templates
Preview how your notification templates will look before using them in your contact points, helping you understand the result of the template you are creating as well as enabling you to fix any errors before saving it.
**Note:** This feature is only for Grafana Alertmanager.
To preview your notification templates:
1. Navigate to **Alerts&IRM** -> **Alerting** -> **Contact points** -> **Notification Templates**.
1. Click **+ Add notification template** or edit an existing template.
1. Add or update your template content.
Default data is provided and you can add or edit alert data to it as well as alert instances. You can add alert data directly in the Payload data window itself or click **Select alert instances** or **Add custom alerts**.
1. [Optional] To add alert data from existing alert instances:
a. Click **Select alert instances**.
b. Hover over the alert instances to view more information on each alert instance.
c. Click **Confirm** to add the alert instance(s) to the payload.
1. [Optional] To add alert data using the Alert data editor, click **Add custom data:**
a. Add annotations, custom labels and/or set a dashboard or a panel.
b. Toggle Firing/resolved depending on whether you want to add firing or resolved alerts to your notification.
c. Click **Add alert data**.
d. Click **Refresh preview** to see what your template content will look like and the corresponding payload data.
If there are any errors in your template, they are displayed in the Preview and you can correct them before saving.
1. Save your changes.
## Template the subject of an email
Template the subject of an email to contain the number of firing and resolved alerts:
```
1 firing alert(s), 0 resolved alerts(s)
```
1. Create a template called `email.subject` with the following content:
```
{{ define "email.subject" }}
{{ len .Alerts.Firing }} firing alert(s), {{ len .Alerts.Resolved }} resolved alert(s)
{{ end }}
```
2. Execute the template from the subject field in your contact point integration:
```
{{ template "email.subject" . }}
```
## Template the message of an email
Template the message of an email to contain a summary of all firing and resolved alerts:
```
There are 2 firing alert(s), and 1 resolved alert(s)
Firing alerts:
- alertname=Test 1 grafana_folder=GrafanaCloud has value(s) B=1
- alertname=Test 2 grafana_folder=GrafanaCloud has value(s) B=2
Resolved alerts:
- alertname=Test 3 grafana_folder=GrafanaCloud has value(s) B=0
```
1. Create a notification template called `email` with two templates in the content: `email.message_alert` and `email.message`.
The `email.message_alert` template is used to print the labels and values for each firing and resolved alert while the `email.message` template contains the structure of the email.
```
{{- define "email.message_alert" -}}
{{- range .Labels.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }} has value(s)
{{- range $k, $v := .Values }} {{ $k }}={{ $v }}{{ end }}
{{- end -}}
{{ define "email.message" }}
There are {{ len .Alerts.Firing }} firing alert(s), and {{ len .Alerts.Resolved }} resolved alert(s)
{{ if .Alerts.Firing -}}
Firing alerts:
{{- range .Alerts.Firing }}
- {{ template "email.message_alert" . }}
{{- end }}
{{- end }}
{{ if .Alerts.Resolved -}}
Resolved alerts:
{{- range .Alerts.Resolved }}
- {{ template "email.message_alert" . }}
{{- end }}
{{- end }}
{{ end }}
```
2. Execute the template from the message field in your contact point integration:
```
{{ template "email.message" . }}
```
## Template the title of a Slack message
Template the title of a Slack message to contain the number of firing and resolved alerts:
```
1 firing alert(s), 0 resolved alerts(s)
```
1. Create a template called `slack.title` with the following content:
```
{{ define "slack.title" }}
{{ len .Alerts.Firing }} firing alert(s), {{ len .Alerts.Resolved }} resolved alert(s)
{{ end }}
```
2. Execute the template from the title field in your contact point integration:
```
{{ template "slack.title" . }}
```
## Template the content of a Slack message
Template the content of a Slack message to contain a description of all firing and resolved alerts, including their labels, annotations, Silence URL and Dashboard URL.
**Note:**
This template is for Grafana-managed alerts only.
To use the template for Grafana Mimir/Loki-managed alerts, delete the references to DashboardURL and SilenceURL.
For more information, see the [Prometheus documentation on notifications](https://prometheus.io/docs/alerting/latest/notifications/).
```
1 firing alert(s):
[firing] Test1
Labels:
- alertname: Test1
- grafana_folder: GrafanaCloud
Annotations:
- description: This is a test alert
Silence: https://example.com/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DTest1&matcher=grafana_folder%3DGrafanaCloud
Go to dashboard: https://example.com/d/dlhdLqF4z?orgId=1
1 resolved alert(s):
[firing] Test2
Labels:
- alertname: Test2
- grafana_folder: GrafanaCloud
Annotations:
- description: This is another test alert
Silence: https://example.com/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DTest2&matcher=grafana_folder%3DGrafanaCloud
Go to dashboard: https://example.com/d/dlhdLqF4z?orgId=1
```
1. Create a template called `slack` with two templates in the content: `slack.print_alert` and `slack.message`.
The `slack.print_alert` template is used to print the labels, annotations, SilenceURL and DashboardURL while the `slack.message` template contains the structure of the notification.
```
{{ define "slack.print_alert" -}}
[{{.Status}}] {{ .Labels.alertname }}
Labels:
{{ range .Labels.SortedPairs -}}
- {{ .Name }}: {{ .Value }}
{{ end -}}
{{ if .Annotations -}}
Annotations:
{{ range .Annotations.SortedPairs -}}
- {{ .Name }}: {{ .Value }}
{{ end -}}
{{ end -}}
{{ if .SilenceURL -}}
Silence: {{ .SilenceURL }}
{{ end -}}
{{ if .DashboardURL -}}
Go to dashboard: {{ .DashboardURL }}
{{- end }}
{{- end }}
{{ define "slack.message" -}}
{{ if .Alerts.Firing -}}
{{ len .Alerts.Firing }} firing alert(s):
{{ range .Alerts.Firing }}
{{ template "slack.print_alert" . }}
{{ end -}}
{{ end }}
{{ if .Alerts.Resolved -}}
{{ len .Alerts.Resolved }} resolved alert(s):
{{ range .Alerts.Resolved }}
{{ template "slack.print_alert" .}}
{{ end -}}
{{ end }}
{{- end }}
```
2. Execute the template from the text body field in your contact point integration:
```
{{ template "slack.message" . }}
```
## Template both email and Slack with shared templates
Instead of creating separate notification templates for email and Slack, you can share the same template.
For example, if you want to send an email with this subject and Slack message with this title:
```
1 firing alert(s), 0 resolved alerts(s)
```
1. Create a template called `common.subject_title` with the following content:
```
{{ define "common.subject_title" }}
{{ len .Alerts.Firing }} firing alert(s), {{ len .Alerts.Resolved }} resolved alert(s)
{{ end }}
```
2. For email, execute the template from the subject field in your email contact point integration:
```
{{ template "common.subject_title" . }}
```
3. For Slack, execute the template from the title field in your Slack contact point integration:
```
{{ template "common.subject_title" . }}
```
docs/sources/alerting/configure-notifications/template-notifications/images-in-notifications.md
-139
View File
@@ -1,139 +0,0 @@
---
aliases:
- ./alerting-rules/manage-notifications/manage-contact-points/template-notifications/images-in-notifications/
canonical: https://grafana.com/docs/grafana/latest/alerting/manage-notifications/images-in-notifications/
description: Use images in notifications to help users better understand why alerts are firing or have been resolved
keywords:
- grafana
- alerting
- images
- notifications
labels:
products:
- cloud
- enterprise
- oss
title: Use images in notifications
weight: 500
---
# Use images in notifications
Images in notifications helps recipients of alert notifications better understand why an alert has fired or resolved by including a screenshot of the panel associated with the alert.
{{% admonition type="note" %}}
This feature is not supported in Mimir or Loki, or when Grafana is configured to send alerts to other Alertmanagers such as the Prometheus Alertmanager
{{% /admonition %}}
When an alert is fired or resolved Grafana takes a screenshot of the panel associated with the alert. This is determined via the Dashboard UID and Panel ID annotations of the rule. Grafana cannot take a screenshot for alerts that are not associated with a panel.
Grafana takes at most two screenshots for each alert: once when the alert fires and again when the alert is resolved. Screenshots are not re-taken over the lifetime of the alert, instead you should open the panel in Grafana to follow the data in real time. In addition, depending on how alerts are grouped in your notification policies, Grafana might send a notification with many screenshots of the same panel. This happens because Grafana does not know how your alerts are grouped at the time a screenshot is taken, and so acts conservatively by taking a screenshot for every alert.
Once a screenshot has been taken Grafana can either upload it to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; upload the screenshot to it's internal web server; or upload it to the service that is receiving the notification, such as Slack. Which option you should choose depends on how your Grafana is managed and which integrations you use. More information on this can be found in Requirements.
Refer to the table at the end of this page for a list of contact points and their support for images in notifications.
## Requirements
1. To use images in notifications, Grafana must be set up to use image rendering. You can either install the image rendering plugin or run it as a remote rendering service.
2. When a screenshot is taken it is saved to the [data][paths] folder, even if Grafana is configured to upload screenshots to a cloud storage service. Grafana must have write-access to this folder otherwise screenshots cannot be saved to disk and an error will be logged for each failed screenshot attempt.
3. You should use a cloud storage service unless sending alerts to Discord, Email, Pushover, Slack or Telegram. These integrations support either embedding screenshots in the email or attaching screenshots to the notification, while other integrations must link screenshots uploaded to a cloud storage bucket. If a cloud storage service has been configured then integrations that support both will link screenshots from the cloud storage bucket instead of embedding or attaching screenshots to the notification.
4. If uploading screenshots to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; and accessing screenshots in the bucket requires authentication, logging into a VPN or corporate network; then image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN. If this happens we recommend using [integrations which support uploading images](#supported-contact-points) or [disabling images in notifications](#configuration) altogether.
5. When uploading screenshots to a cloud storage service Grafana uses a random 20 character (30 characters for Azure Blob Storage) filename for each image. This makes URLs hard to guess but not impossible.
6. Grafana does not delete screenshots from cloud storage. We recommend configuring a retention policy with your cloud storage service to delete screenshots older than 1 month.
7. If Grafana is configured to upload screenshots to its internal web server, and accessing Grafana requires logging into a VPN or corporate network; image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN. If this happens we recommend using [integrations which support uploading images](#supported-contact-points) or [disabling images in notifications](#configuration) altogether.
8. Grafana does not delete screenshots uploaded to its internal web server. To delete screenshots from `static_root_path/images/attachments` after a certain amount of time we recommend setting up a CRON job.
## Configuration
{{% admonition type="note" %}}
Grafana Cloud users can request this feature by [opening a support ticket in the Cloud Portal](/profile/org#support).
{{% /admonition %}}
Having installed either the image rendering plugin, or set up Grafana to use a remote rendering service, set `capture` in `[unified_alerting.screenshots]` to `true`:
# Enable screenshots in notifications. You must have either installed the Grafana image rendering
# plugin, or set up Grafana to use a remote rendering service.
# For more information on configuration options, refer to [rendering].
capture = false
If screenshots should be uploaded to cloud storage then `upload_external_image_storage` should also be set to `true`:
# Uploads screenshots to the local Grafana server or remote storage such as Azure, S3 and GCS. Please
# see [external_image_storage] for further configuration options. If this option is false, screenshots
# will be persisted to disk for up to temp_data_lifetime.
upload_external_image_storage = false
Restart Grafana for the changes to take effect.
## Advanced configuration
We recommended that `max_concurrent_screenshots` is less than or equal to `concurrent_render_request_limit`. The default value for both `max_concurrent_screenshots` and `concurrent_render_request_limit` is `5`:
# The maximum number of screenshots that can be taken at the same time. This option is different from
# concurrent_render_request_limit as max_concurrent_screenshots sets the number of concurrent screenshots
# that can be taken at the same time for all firing alerts where as concurrent_render_request_limit sets
# the total number of concurrent screenshots across all Grafana services.
max_concurrent_screenshots = 5
## Supported contact points
Grafana supports a wide range of contact points with varied support for images in notifications. The table below shows the list of all contact points supported in Grafana and their support for uploading screenshots to the receiving service and referencing screenshots that have been uploaded to a cloud storage service.
| Name | Upload from disk | Reference from cloud storage |
| ----------------------- | ---------------------------------------------------------- | -------------------------------------------------------- |
| DingDing | No | No |
| Discord | Yes (Maximum of 10 per notification) | Yes (Maximum of 10 per notification) |
| Email | Yes (Embedded in the email) | Yes |
| Google Chat | No | Yes |
| Kafka | No | No |
| Line | No | No |
| Microsoft Teams | No | Yes |
| Opsgenie | No | Yes |
| Pagerduty | No | Yes |
| Prometheus Alertmanager | No | No |
| Pushover | Yes (Maximum of 1 per notification) | No |
| Sensu Go | No | No |
| Slack | Yes (when using Bot tokens, maximum of 5 per notification) | Yes (when using webhooks, maximum of 1 per notification) |
| Telegram | Yes | No |
| Threema | No | No |
| VictorOps | No | No |
| Webhook | No | Yes |
## Limitations
- This feature is not supported in Mimir or Loki, or when Grafana is configured to send alerts to other Alertmanagers such as the Prometheus Alertmanager.
- A number of contact points support at most one image per notification. In this case, just the first image is either uploaded to the receiving service or referenced from cloud storage per notification.
- When multiple alerts are sent in a single notification a screenshot might be included for each alert. The order the images are shown is random.
- If uploading screenshots to a cloud storage service such as Amazon S3, Azure Blob Storage or Google Cloud Storage; and accessing screenshots in the bucket requires authentication, logging into a VPN or corporate network; image previews might not work in all instant messaging and communication platforms as some services rewrite URLs to use their CDN.
## Troubleshooting
If Grafana has been set up to send images in notifications, however notifications are still being received without them, follow the troubleshooting steps below:
1. Check that images in notifications has been set up as per the instructions.
2. Enable debug logging in Grafana and look for logs with the logger `ngalert.image`.
3. If the alert is not associated with a dashboard there will be logs for `Cannot take screenshot for alert rule as it is not associated with a dashboard`.
4. If the alert is associated with a dashboard, but no panel in the dashboard, there will be logs for `Cannot take screenshot for alert rule as it is not associated with a panel`.
5. If images cannot be taken because of mis-configuration or an issue with image rendering there will be logs for `Failed to take an image` including the Dashboard UID, Panel ID, and the error message.
6. Check that the contact point supports images in notifications and whether it supports uploading images to the receiving service or referencing images that have been uploaded to a cloud storage service.
## Metrics
Grafana provides the following metrics to observe the performance and failure rate of images in notifications.
For example, if a screenshot could not be taken within the expected time (10 seconds) then the counter `grafana_screenshot_failures_total` is updated.
- `grafana_alerting_image_cache_hits_total`
- `grafana_alerting_image_cache_misses_total`
- `grafana_screenshot_duration_seconds`
- `grafana_screenshot_failures_total`
- `grafana_screenshot_successes_total`
- `grafana_screenshot_upload_failures_total`
- `grafana_screenshot_upload_successes_total`
docs/sources/alerting/configure-notifications/template-notifications/reference.md
-83
View File
@@ -1,83 +0,0 @@
---
aliases:
- ./alerting-rules/manage-notifications/manage-contact-points/template-notifications/reference/
canonical: https://grafana.com/docs/grafana/latest/alerting/manage-notifications/template-notifications/reference/
description: Learn about templating notifications options
keywords:
- grafana
- alerting
- notifications
- templates
labels:
products:
- cloud
- enterprise
- oss
title: Reference
weight: 400
---
# Reference
## Data
### Alert
| Name | Kind | Description | Example |
| ------------ | -------- | ------------------------------------------------------------------------------------ | --------------------- |
| Status | `string` | Firing or resolved | `{{ .Status }}` |
| Labels | `KV` | The labels for this alert | `{{ .Labels }}` |
| Annotations | `KV` | The annotations for this alert | `{{ .Annotations }}` |
| Values | `KV` | The values of all expressions, including Classic Conditions | `{{ .Values }}` |
| StartsAt | `Time` | The time the alert fired | `{{ .StartsAt }}` |
| EndsAt | `Time` | | `{{ .EndsAt }}` |
| GeneratorURL | `string` | A link to Grafana, or the Alertmanager if using an external Alertmanager | `{{ .GeneratorURL }}` |
| SilenceURL | `string` | A link to silence the alert | `{{ .SilenceURL }}` |
| DashboardURL | `string` | A link to the Grafana Dashboard if the alert has a Dashboard UID annotation | `{{ .DashboardURL }}` |
| PanelURL | `string` | A link to the panel if the alert has a Panel ID annotation | `{{ .PanelURL }}` |
| Fingerprint | `string` | A unique string that identifies the alert | `{{ .Fingerprint }}` |
| ValueString | `string` | A string that contains the labels and value of each reduced expression in the alert. | `{{ .ValueString }}` |
### ExtendedData
| Name | Kind | Description | Example |
| ----------------- | --------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
| Receiver | `string` | The name of the contact point sending the notification | `{{ .Receiver }}` |
| Status | `string` | The status is `firing` if at least one alert is firing, otherwise `resolved` | `{{ .Status }}` |
| Alerts | `[]Alert` | List of all firing and resolved alerts in this notification | `There are {{ len .Alerts }} alerts` |
| Firing alerts | `[]Alert` | List of all firing alerts in this notification | `There are {{ len .Alerts.Firing }} firing alerts` |
| Resolved alerts | `[]Alert` | List of all resolved alerts in this notification | `There are {{ len .Alerts.Resolved }} resolved alerts` |
| GroupLabels | `KV` | The labels that group these alerts in this | `{{ .GroupLabels }}` |
| CommonLabels | `KV` | The labels common to all alerts in this notification | `{{ .CommonLabels }}` |
| CommonAnnotations | `KV` | The annotations common to all alerts in this notification | `{{ .CommonAnnotations }}` |
| ExternalURL | `string` | A link to Grafana, or the Alertmanager that sent this notification if using an external Alertmanager | `{{ .ExternalURL }}` |
### KV
`KV` is a set of key value pairs, where each key and value is a string. If a KV happens to contain numbers or bools then these are string representations of the numeric or boolean value.
Here is an example of a KV, the annotations of an alert:
```yaml
summary: 'A summary of the alert'
description: 'A description of the alert'
```
In addition to iterating over each key value pair, you can sort the pairs, remove keys, and iterate over just the keys or the values.
| Name | Description | Arguments | Returns | Example |
| ----------- | ---------------------------------------------- | --------- | ------- | ------------------------------------- |
| SortedPairs | Sorts | | | `{{ .Annotations.SortedPairs }}` |
| Remove | Returns a copy of the KV with the keys removed | []string | | `{{ .Annotations.Remove "summary" }}` |
| Names | A list of the names | | | `{{ .Names }}` |
| Values | A list of the values | | | `{{ .Values }}` |
### Time
Time is from the Go [`time`](https://pkg.go.dev/time#Time) package. You can print a time in a number of different formats. For example, to print the time that an alert fired in the format `Monday, 1st January 2022 at 10:00AM` you would write the following template:
```
{{ .StartsAt.Format "Monday, 2 January 2006 at 3:04PM" }}
```
You can find a reference for Go's time format [here](https://pkg.go.dev/time#pkg-constants).
docs/sources/alerting/configure-notifications/template-notifications/use-notification-templates.md
-45
View File
@@ -1,45 +0,0 @@
---
aliases:
- ./alerting-rules/manage-notifications/manage-contact-points/template-notifications/use-notification-templates/
canonical: https://grafana.com/docs/grafana/latest/alerting/manage-notifications/template-notifications/use-notification-templates/
description: Use notification templates in contact points to customize your notifications
keywords:
- grafana
- alerting
- notifications
- templates
- use templates
labels:
products:
- cloud
- enterprise
- oss
title: Use notification templates
weight: 300
---
# Use notification templates
Use templates in contact points to customize your notifications.
In the Contact points tab, you can see a list of your contact points.
1. To create a new contact point, click New.
**Note:** You can edit an existing contact by clicking the Edit icon.
1. Execute a template from one or more fields such as Message and Subject:
{{< figure max-width="940px" src="/static/img/docs/alerting/unified/use-notification-template-9-4.png" caption="Use notification template" >}}
For more information on how to write and execute templates, refer to [Using Go's templating language][using-go-templating-language] and [Create notification templates][create-notification-templates].
1. Click **Save contact point**.
{{% docs/reference %}}
[create-notification-templates]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/create-notification-templates"
[create-notification-templates]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/create-notification-templates"
[using-go-templating-language]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/using-go-templating-language"
[using-go-templating-language]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/using-go-templating-language"
{{% /docs/reference %}}
docs/sources/alerting/configure-notifications/template-notifications/using-go-templating-language.md
-293
View File
@@ -1,293 +0,0 @@
---
aliases:
- ./alerting-rules/manage-notifications/manage-contact-points/template-notifications/using-go-templating-language/
canonical: https://grafana.com/docs/grafana/latest/alerting/manage-notifications/template-notifications/using-go-templating-language/
description: Use Go's templating language to create your own notification templates
keywords:
- grafana
- alerting
- notifications
- templates
- write templates
labels:
products:
- cloud
- enterprise
- oss
title: Using Go's templating language
weight: 100
---
# Using Go's templating language
You write notification templates in Go's templating language, [text/template](https://pkg.go.dev/text/template).
Before you start creating your own notification templates, we recommend that you read through this topic, which provides you with an overview of Go's templating language and writing templates in text/template.
## Dot
In text/template there is a special cursor called dot, and is written as `.`. You can think of this cursor as a variable whose value changes depending where in the template it is used. For example, at the start of a notification template `.` refers to something called [`ExtendedData`][extendeddata] which contains a number of fields including `Alerts`, `Status`, `GroupLabels`, `CommonLabels`, `CommonAnnotations` and `ExternalURL`. However, dot might refer to something else when used in a range over a list, when used inside a `with`, or when writing feature templates to be used in other templates. You can see examples of this in [Create notification templates][create-notification-templates], and all data and functions in the [Reference][reference].
## Opening and closing tags
In text/template, templates start with `{{` and end with `}}` irrespective of whether the template prints a variable or executes control structures such as if statements. This is different from other templating languages such as Jinja where printing a variable uses `{{` and `}}` and control structures use `{%` and `%}`.
## Print
To print the value of something use `{{` and `}}`. You can print the value of dot, a field of dot, the result of a function, and the value of a [variable](#variables). For example, to print the `Alerts` field where dot refers to `ExtendedData` you would write the following:
```
{{ .Alerts }}
```
## Iterate over alerts
To print just the labels of each alert, rather than all information about the alert, you can use a `range` to iterate the alerts in `ExtendedData`:
```
{{ range .Alerts }}
{{ .Labels }}
{{ end }}
```
Inside the range dot no longer refers to `ExtendedData`, but to an `Alert`. You can use `{{ .Labels }}` to print the labels of each alert. This works because `{{ range .Alerts }}` changes dot to refer to the current alert in the list of alerts. When the range is finished dot is reset to the value it had before the start of the range, which in this example is `ExtendedData`:
```
{{ range .Alerts }}
{{ .Labels }}
{{ end }}
{{/* does not work, .Labels does not exist here */}}
{{ .Labels }}
{{/* works, cursor was reset */}}
{{ .Status }}
```
## Iterate over annotations and labels
Let's write a template to print the labels of each alert in the format `The name of the label is $name, and the value is $value`, where `$name` and `$value` contain the name and value of each label.
Like in the previous example, use a range to iterate over the alerts in `.Alerts` such that dot refers to the current alert in the list of alerts, and then use a second range on the sorted labels so dot is updated a second time to refer to the current label. Inside the second range use `.Name` and `.Value` to print the name and value of each label:
```
{{ range .Alerts }}
{{ range .Labels.SortedPairs }}
The name of the label is {{ .Name }}, and the value is {{ .Value }}
{{ end }}
{{ range .Annotations.SortedPairs }}
The name of the annotation is {{ .Name }}, and the value is {{ .Value }}
{{ end }}
{{ end }}
```
## The index function
To print a specific annotation or label use the `index` function.
```
{{ range .Alerts }}
The name of the alert is {{ index .Labels "alertname" }}
{{ end }}
```
## If statements
You can use if statements in templates. For example, to print `There are no alerts` if there are no alerts in `.Alerts` you would write the following:
```
{{ if .Alerts }}
There are alerts
{{ else }}
There are no alerts
{{ end }}
```
## With
With is similar to if statements, however unlike if statements, `with` updates dot to refer to the value of the with:
```
{{ with .Alerts }}
There are {{ len . }} alert(s)
{{ else }}
There are no alerts
{{ end }}
```
## Variables
Variables in text/template must be created within the template. For example, to create a variable called `$variable` with the current value of dot you would write the following:
```
{{ $variable := . }}
```
You can use `$variable` inside a range or `with` and it will refer to the value of dot at the time the variable was defined, not the current value of dot.
For example, you cannot write a template that use `{{ .Labels }}` in the second range because here dot refers to the current label, not the current alert:
```
{{ range .Alerts }}
{{ range .Labels.SortedPairs }}
{{ .Name }} = {{ .Value }}
{{/* does not work because in the second range . is a label not an alert */}}
There are {{ len .Labels }}
{{ end }}
{{ end }}
```
You can fix this by defining a variable called `$alert` in the first range and before the second range:
```
{{ range .Alerts }}
{{ $alert := . }}
{{ range .Labels.SortedPairs }}
{{ .Name }} = {{ .Value }}
{{/* works because $alert refers to the value of dot inside the first range */}}
There are {{ len $alert.Labels }}
{{ end }}
{{ end }}
```
## Range with index
You can get the index of each alert within a range by defining index and value variables at the start of the range:
```
{{ $num_alerts := len .Alerts }}
{{ range $index, $alert := .Alerts }}
This is alert {{ $index }} out of {{ $num_alerts }}
{{ end }}
```
## Define templates
You can define templates using `define` and the name of the template in double quotes. You should not define templates with the same name as other templates, including default templates such as `__subject`, `__text_values_list`, `__text_alert_list`, `default.title` and `default.message`. Where a template has been created with the same name as a default template, or a template in another notification template, Grafana might use either template. Grafana does not prevent, or show an error message, when there are two or more templates with the same name.
```
{{ define "print_labels" }}
{{ end }}
```
## Execute templates
You can execute defined templates using `template`, the name of the template in double quotes, and the cursor that should be passed to the template:
```
{{ template "print_labels" . }}
```
## Pass data to templates
Within a template dot refers to the value that is passed to the template.
For example, if a template is passed a list of firing alerts then dot refers to that list of firing alerts:
```
{{ template "print_alerts" .Alerts }}
```
If the template is passed the sorted labels for an alert then dot refers to the list of sorted labels:
```
{{ template "print_labels" .SortedLabels }}
```
This is useful when writing reusable templates. For example, to print all alerts you might write the following:
```
{{ template "print_alerts" .Alerts }}
```
Then to print just the firing alerts you could write this:
```
{{ template "print_alerts" .Alerts.Firing }}
```
This works because both `.Alerts` and `.Alerts.Firing` is a list of alerts.
```
{{ define "print_alerts" }}
{{ range . }}
{{ template "print_labels" .SortedLabels }}
{{ end }}
{{ end }}
```
## Comments
You can add comments with `{{/*` and `*/}}`:
```
{{/* This is a comment */}}
```
To prevent comments from adding line breaks use:
```
{{- /* This is a comment with no leading or trailing line breaks */ -}}
```
## Indentation
You can use indentation, both tabs and spaces, and line breaks, to make templates more readable:
```
{{ range .Alerts }}
{{ range .Labels.SortedPairs }}
{{ .Name }} = {{ .Value }}
{{ end }}
{{ end }}
```
However, indentation in the template will also be present in the text. Next we will see how to remove it.
## Remove spaces and line breaks
In text/template use `{{-` and `-}}` to remove leading and trailing spaces and line breaks.
For example, when using indentation and line breaks to make a template more readable:
```
{{ range .Alerts }}
{{ range .Labels.SortedPairs }}
{{ .Name }} = {{ .Value }}
{{ end }}
{{ end }}
```
The indentation and line breaks will also be present in the text:
```
alertname = "Test"
grafana_folder = "Test alerts"
```
You can remove the indentation and line breaks from the text changing `}}` to `-}}` at the start of each range:
```
{{ range .Alerts -}}
{{ range .Labels.SortedPairs -}}
{{ .Name }} = {{ .Value }}
{{ end }}
{{ end }}
```
The indentation and line breaks in the template are now absent from the text:
```
alertname = "Test"
grafana_folder = "Test alerts"
```
{{% docs/reference %}}
[create-notification-templates]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/create-notification-templates"
[create-notification-templates]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/create-notification-templates"
[extendeddata]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference#extendeddata"
[extendeddata]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference#extendeddata"
[reference]: "/docs/grafana/ -> /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/template-notifications/reference"
[reference]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference"
{{% /docs/reference %}}