diff --git a/docs/sources/alerting/alerting-rules/templates/_index.md b/docs/sources/alerting/alerting-rules/templates/_index.md new file mode 100644 index 00000000000..77ef6073989 --- /dev/null +++ b/docs/sources/alerting/alerting-rules/templates/_index.md @@ -0,0 +1,215 @@ +--- +aliases: + - ../fundamentals/annotation-label/variables-label-annotation/ # /docs/grafana//alerting/fundamentals/annotation-label/variables-label-annotation/ + - ../alerting-rules/templating-labels-annotations/ # /docs/grafana//alerting-rules/templating-labels-annotations/ +canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/ +description: Learn how to template annotations and labels to include data from queries and expressions in alert messages +keywords: + - grafana + - alerting + - templating + - labels + - annotations +labels: + products: + - cloud + - enterprise + - oss +title: Template annotations and labels +weight: 500 +refs: + labels: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/fundamentals/alert-rules/annotation-label/#labels + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/#labels + values: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/fundamentals/alert-rules/annotation-label/#values + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/#values + annotations: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/fundamentals/alert-rules/annotation-label/#annotations + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/#annotations + explore: + - pattern: /docs/ + destination: /docs/grafana//explore/ + intro-to-templates: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/fundamentals/templates/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/templates/ + alert-rule-template-reference: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/reference/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/ + alert-rule-template-examples: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/examples/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/examples/ + notification-template-reference: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/ + notification-data-reference: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/#notification-data + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/#notification-data + view-alert-state: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/manage-notifications/view-state-health/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/manage-notifications/view-state-health/ + preview-notifications: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/manage-notification-templates/#preview-notification-templates + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/manage-notification-templates/#preview-notification-templates +--- + +# Template annotations and labels + +You can use templates to customize alert and notification messages, including dynamic data from alert rule queries. + +In Grafana Alerting, you can template alert messages in two ways. + +1. **Template annotations and labels**: In the alert rule definition, you can template annotations and labels to include extra information from query data to the alert, adding meaningful details based on the query results. +1. **Template notifications**: You can template notifications to control the content and appearance of your notifications. + +## How templating works + +In this diagram, you can see the differences between both types of templates. + +{{< figure src="/media/docs/alerting/how-notification-templates-works.png" max-width="1200px" alt="How templating works" >}} + +Refer to [Templates Introduction](ref:intro-to-templates) for a more detailed explanation of this diagram. + +Both types of templates are written in the Go templating system. However, it's important to understand that variables and functions used in notification templates are different from those used in annotation and label templates. + +1. **Template annotations and labels**: These templates add extra information to individual alert instances. Template variables like [`$labels`](ref:labels) and [`$values`](ref:values) represent alert query data of the individual alert instance. +1. **Template notifications**: Notification templates format the notification content for a group of alerts. Variables like [`.Alerts`](ref:notification-data-reference) include all firing and resolved alerts in the notification. + +## Template annotations + +[Annotations](ref:annotations) add additional information to alert instances and are often used to help identify the alert and guide responders on how to address the issue. + +Annotations are key-value pairs defined in the alert rule. They can contain plain text or template code that is evaluated when the alert fires. + +Grafana includes several optional annotations, such as `description`, `summary`, `runbook_url`, `dashboardUId` and `panelId`, which can be edited in the alert rule. You can also create your custom annotations. For example, you might create a new annotation named `location` to report the location of the system that triggered the alert. + +Here’s an example of a `summary` annotation explaining why the alert was triggered, using plain text. + +``` +CPU usage has exceeded 80% for the last 5 minutes. +``` + +However, if you want to display dynamic query values in annotations, you need to use template code. Common use cases include: + +- Displaying the query value or threshold that triggered the alert. +- Highlighting label information that identifies the alert, such as environment, region, or priority. +- Providing specific instructions based on query values. +- Customizing runbook links depending on query or label values. +- Including contact information based on alert labels. + +For instance, you can template the previous example to display the specific instance and CPU value that triggered the alert. + +``` +CPU usage for {{ $labels.instance }} has exceeded 80% ({{ $values.A.Value }}) for the last 5 minutes. +``` + +Alternatively, you can use the `index` function to print query values. + +``` +CPU usage for {{ index $labels "instance" }} has exceeded 80% ({{ index $values "A" }}) for the last 5 minutes. +``` + +The result of the annotation would be as follows. + +``` +CPU usage for Instance 1 has exceeded 80% (81.2345) for the last 5 minutes. +``` + +### How to template an annotation + +Complete the following steps to template an annotation. + +1. Navigate to **Alerts & IRM** -> **Alert rules** -> create or edit an alert rule. +1. Scroll down to the **Configure notification message** section. +1. Copy in your template in the corresponding annotation field (`summary`, `description`, `runbook_url`, `custom`). + +### Preview annotation templates + +You can template annotations when creating or editing an alert rule. + +{{< figure src="/media/docs/alerting/alert-rule-using-annotation-template.png" max-width="1200px" alt="An alert rule templating the annotation summary" >}} + +Two common methods are used to test and preview annotation templates: + +1. Trigger the alert and [view the alert instance state in the Grafana UI](ref:view-alert-state), where all annotations of the alert instance are displayed. +1. Use a notification template that displays all annotations, then [preview the notification template](ref:preview-notifications) using the alert instance. + +## Template labels + +The set of [labels](ref:labels) for an alert instance is used to uniquely identify that alert among all other alert instances. + +Labels determine how alerts are routed and managed for notifications, making their design key to the effectiveness of your alerting system. + +Labels can be returned from an alert rule query, such as the `pod` label in a Kubernetes Prometheus query. You can also define additional labels in the alert rule to provide extra information for processing alerts. + +Like annotations, labels are key-value pairs that can contain plain text or template code evaluated when the alert fires. + +Template labels when the labels returned by your queries are insufficient. For instance: + +- A new label based on a query value can group a subset of alerts differently, changing how notifications are sent. +- A new label based on a query value can be used in a notification policy to alter the notification contact point. + +Here’s an example of templating a `severity` label based on the query value. + +``` +{{ if (gt $values.A.Value 90.0) -}} +critical +{{ else if (gt $values.A.Value 80.0) -}} +high +{{ else if (gt $values.A.Value 60.0) -}} +medium +{{ else -}} +low +{{- end }} +``` + +In this example, the value of the `severity` label is determined by the query value, and the possible options are `critical`, `high`, `medium`, or `low`. You can then use the `severity` label to change their notifications—for instance, sending `critical` alerts immediately or routing `low` alerts to a specific team for further review. + +{{% admonition type="note" %}} +You should avoid displaying query values in labels, as this may create numerous unique alert instances—one for each distinct label value. Instead, use annotations for query values. +{{% /admonition %}} + +### How to template a label + +Complete the following steps to template a label. + +1. Navigate to **Alerts & IRM** -> **Alert rules** -> create or edit an alert rule. +1. Scroll down to the **Configure labels and notifications** section. +1. Click **+ Add labels**. +1. Enter a **key** that identifies the label. +1. Copy in your template in the **value** field. + +### Preview label templates + +You can template label values when creating or editing an alert rule. + +To preview label values, select `Use notification policy`, and then click on `Preview routing`. + +{{< figure src="/media/docs/alerting/alert-instance-routing-preview.png" max-width="1200px" alt="Routing preview displays label values" >}} + +## More information + +For further details on how to template alert rules, refer to: + +- [Annotation and label template reference](ref:alert-rule-template-reference) +- [Annotation and label examples](ref:alert-rule-template-examples) diff --git a/docs/sources/alerting/alerting-rules/templates/examples.md b/docs/sources/alerting/alerting-rules/templates/examples.md new file mode 100644 index 00000000000..47e562544a2 --- /dev/null +++ b/docs/sources/alerting/alerting-rules/templates/examples.md @@ -0,0 +1,308 @@ +--- +canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/examples/ +description: Examples of templating labels and annotations in Grafana alert rules +keywords: + - grafana + - alerting + - templating + - labels + - annotations +labels: + products: + - cloud + - enterprise + - oss +title: Labels and annotations template examples +menuTitle: Examples +weight: 102 +refs: + labels: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/fundamentals/alert-rules/annotation-label/#labels + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/#labels + annotations: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/fundamentals/alert-rules/annotation-label/#annotations + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/#annotations + alert-rule-templates: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/ + - pattern: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/ + reference: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/reference/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/ + reference-labels: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/reference/#labels + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/#labels + reference-values: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/reference/#values + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/#values + reference-humanize: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/reference/#humanize + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/#humanize + reference-humanizepercentage: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/reference/#humanizepercentage + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/#humanizepercentage + reference-match: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/reference/#match + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/#match + reference-functions: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/reference/#functions + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/#functions + language-functions: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/language/#functions + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/language/#functions + language-index: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/language/#functions + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/language/#functions + language: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/language + - pattern: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/language +--- + +# Labels and annotations template examples + +Templating allows you to add dynamic data from queries to alert labels and annotations. Dynamic data enhances alert context, making it easier for responders to quickly assess and address the issue. + +This page provides common examples for templating labels and annotations. For more information on templating, refer to: + +- [Template annotations and labels](ref:alert-rule-templates) +- [Annotation and label template reference](ref:reference) +- [Alerting template language](ref:language) + +## Annotation example + +[Annotations](ref:annotations) add extra details to alert instances and are often used to provide helpful information for identifying the issue and guiding the response. + +A common use case for annotations is to display the specific query value or threshold that triggered the alert. + +For example, you can display the query value from the [`$values`](ref:reference-values) variable to inform about the CPU value that triggered the alert. + +``` +CPU usage has exceeded 80% ({{ $values.A.value }}) for the last 5 minutes. +``` + +Alternatively, you can use the [`index()`](ref:language-index) function to retrieve the query value as follows. + +``` +CPU usage has exceeded 80% ({{ index $values "A" }}) for the last 5 minutes. +``` + +```template_output +CPU usage has exceeded 80% (81.2345) for the last 5 minutes. +``` + +### Include labels for extra details + +To provide additional context, you can include labels from the query. For instance, access the [`$labels`](ref:reference-labels) variable to display a label that informs about the affected instance: + +``` +CPU usage for {{ $labels.instance }} has exceeded 80% ({{ $values.A.Value }}) for the last 5 minutes. +``` + +```template_output +CPU usage for Instance 1 has exceeded 80% (81.2345) for the last 5 minutes. +``` + +Annotations can also be used to provide a summary of key alert labels, such as the environment and alert severity. For instance, you can display a summary of the alert with important labels like: + +``` +Alert triggered in {{ $labels.environment }} with severity {{ $labels.severity }} +``` + +```template_output +Alert triggered in production with severity critical. +``` + +### Print a range query + +To print the value of an instant query you can print its Ref ID using the `index` function or the `$values` variable: + +``` +{{ $values.A.Value }} +``` + +For range queries, reduce them from a time series to an instant vector using a reduce expression. You can then print the result by referencing its Ref ID. For example, if the reduce expression averages `A` with the Ref ID `B`, you would then print `$values.B`: + +``` +{{ $values.B.Value }} +``` + +### Humanize the value of a query + +To print the humanized value of an instant query, use the [`humanize`](ref:reference-humanize) function: + +``` +{{ humanize $values.A.Value }} +``` + +Alternatively: + +``` +{{ humanize (index $values "A").Value }} +``` + +```template_output +554.9 +``` + +To print the value of an instant query as a percentage, use the [`humanizePercentage`](ref:reference-humanizepercentage) function: + +``` +{{ humanizePercentage $values.A.Value }} +``` + +```template_output +10% +``` + +For additional functions to display or format data, refer to: + +- [Annotation and label template functions](ref:reference-functions) +- [Template language functions](ref:language-functions) + +## Label example + +[Labels](ref:labels) determine how alerts are routed and managed, ensuring that notifications reach the right teams at the right time. If the labels returned by your queries don’t fully capture the necessary context, you can create a new label and sets its value based on query data. + +### Based on query value + +Here’s an example of creating a `severity` label based on a query value: + +```go +{{ if (gt $values.A.Value 90.0) -}} +critical +{{ else if (gt $values.A.Value 80.0) -}} +high +{{ else if (gt $values.A.Value 60.0) -}} +medium +{{ else -}} +low +{{- end }} +``` + +In this example, the `severity` label is determined by the query value: + +- `critical` for values above 90, +- `high` for values above 80, +- `medium` for values above 60, +- and `low` for anything below. + +You can then use the `severity` label to control how alerts are handled. For instance, you could send `critical` alerts immediately, while routing `low` severity alerts to a team for further investigation. + +{{% admonition type="note" %}} +You should avoid displaying query values in labels, as this may create many alert instances—one for each distinct label value. Instead, use annotations to convey query values. +{{% /admonition %}} + +### Based on query label + +You can use labels to differentiate alerts coming from various environments (e.g., production, staging, dev). For example, you may want to add a label that sets the environment based on the instance’s label. Here’s how you can template it: + +```go +{{ if eq $labels.instance "prod-server-1" }}production +{{ else if eq $labels.instance "staging-server-1" }}staging +{{ else }}development +{{ end }} +``` + +This would print: + +- For instance `prod-server-1`, the label would be `production`. +- For `staging-server-1`, the label would be `staging`. +- All other instances would be labeled `development`. + +To make this template more flexible, you can use a regular expression that matches the instance name with the instance name prefix using the [`match()`](ref:reference-match) function: + +```go +{{ if match "^prod-server-.*" $labels.instance }}production +{{ else if match "^staging-server-.*" $labels.instance}}staging +{{ else }}development +{{ end }} +``` + +{{< collapse title="Legacy Alerting templates" >}} + +## Legacy Alerting templates + +For users working with Grafana's legacy alerting system, templates can still be utilized to extract useful information from alert conditions. However, it's important to note that you cannot use `$labels` to print labels from the query if you are using classic conditions, and must use `$values` instead. The reason for this is classic conditions discard these labels to enforce uni-dimensional behavior (at most one alert per alert rule). If classic conditions didn't discard these labels, then queries that returned many time series would cause alerts to flap between firing and resolved constantly as the labels would change every time the alert rule was evaluated. + +Instead, the `$values` variable contains the reduced values of all time series for all conditions that are firing. For example, if you have an alert rule with a query A that returns two time series, and a classic condition B with two conditions, then `$values` would contain `B0`, `B1`, `B2` and `B3`. If the classic condition B had just one condition, then `$values` would contain just `B0` and `B1`. + +#### Print all labels from a classic condition + +To print all labels of all firing time series use the following template (make sure to replace `B` in the regular expression with the Ref ID of the classic condition if it's different): + +```go +{{ range $k, $v := $values -}} +{{ if (match "B[0-9]+" $k) -}} +{{ $k }}: {{ $v.Labels }}{{ end }} +{{ end }} +``` + +For example, a classic condition for two time series exceeding a single condition would print: + +``` +B0: instance=server1 +B1: instance=server2 +``` + +If the classic condition has two or more conditions, and a time series exceeds multiple conditions at the same time, then its labels will be duplicated for each condition that is exceeded: + +``` +B0: instance=server1 +B1: instance=server2 +B2: instance=server1 +B3: instance=server2 +``` + +If you need to print unique labels you should consider changing your alert rules from uni-dimensional to multi-dimensional instead. You can do this by replacing your classic condition with reduce and math expressions. + +#### Print all values from a classic condition + +To print all values from a classic condition take the previous example and replace `$v.Labels` with `$v.Value`: + +```go +{{ range $k, $v := $values -}} +{{ if (match "B[0-9]+" $k) -}} +{{ $k }}: {{ $v.Value }}{{ end }} +{{ end }} +``` + +For example, a classic condition for two time series exceeding a single condition would print: + +``` +B0: 81.2345 +B1: 84.5678 +``` + +If the classic condition has two or more conditions, and a time series exceeds multiple conditions at the same time, then `$values` will contain the values of all conditions: + +``` +B0: 81.2345 +B1: 92.3456 +B2: 84.5678 +B3: 95.6789 +``` + +{{< /collapse >}} diff --git a/docs/sources/alerting/alerting-rules/templates/language.md b/docs/sources/alerting/alerting-rules/templates/language.md new file mode 100644 index 00000000000..a37fabef84f --- /dev/null +++ b/docs/sources/alerting/alerting-rules/templates/language.md @@ -0,0 +1,81 @@ +--- +canonical: https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/language/ +description: Use Go template language to create your notification and alert rule templates +keywords: + - grafana + - alerting + - templates + - write templates +labels: + products: + - cloud + - enterprise + - oss +title: Alerting template language +menuTitle: Template language +refs: + alert-rule-template-reference: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/reference/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/ + alert-rule-template-reference-variables: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/reference/#variables + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/#variables + notification-template-reference: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/ + reference-notificationdata: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/#notification-data + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/#notification-data +--- + +# Alerting template language + +Notification templates and alert rule templates, such as annotations and labels, both use the Go template language, [text/template](https://pkg.go.dev/text/template). + +Both types of templates can use the same keywords, functions, and comparison operators of the Go template language, such as `range`, `if`, `and`, `index`, `eq`, and more. + +However, it's important to note that because notifications and alert rules operate in distinct contexts, some additional variables and functions are only available for either notification or alert rule templates. Refer to: + +- [Annotation and label template reference](ref:alert-rule-template-reference) +- [Notification template reference](ref:notification-template-reference) + +This documentation provides an overview of the functions and operators of the Go template language that are available for both notification and alert rule templates. + +## Print + +To print the value of something, use `{{` and `}}`. You can print the value of a [variable](#variables), a field of a variable, the result of a function, or the value of dot. + +```go +{{ $values }} +{{ $values.A.Value }} +{{ humanize 1000.0 }} +{{ .Alerts }} +``` + +## Dot + +In `text/template`, there is a special cursor called dot, written as `.`. You can think of this cursor as a variable whose value changes depending on where in the template it is used. + +At the start of notification templates, dot (`.`) refers to [Notification Data](ref:reference-notificationdata). + +```go +{{ .Alerts }} +``` + +In annotation and label templates, dot (`.`) is initialized with all alert data. It’s recommended to use the [`$labels` and `$values` variables](ref:alert-rule-template-reference-variables) instead to directly access the alert labels and query values. + +{{% admonition type="note" %}} +Dot (`.`) might refer to something else when used in a [range](#range), a [with](#with), or when writing [templates](#templates) used in other templates. +{{% /admonition %}} + +[//]: <> (The above section is not included in the shared file because `refs` links are not supported in shared files.) + +{{< docs/shared lookup="alerts/template-language.md" source="grafana" version="" >}} diff --git a/docs/sources/alerting/alerting-rules/templates/reference.md b/docs/sources/alerting/alerting-rules/templates/reference.md new file mode 100644 index 00000000000..f33a8632cb3 --- /dev/null +++ b/docs/sources/alerting/alerting-rules/templates/reference.md @@ -0,0 +1,445 @@ +--- +canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/reference/ +description: Reference for variables and functions in Grafana alert rule templating. +keywords: + - grafana + - alerting + - templating + - labels + - annotations +labels: + products: + - cloud + - enterprise + - oss +title: Annotation and label template reference +menuTitle: Template reference +weight: 101 +refs: + notification-template-reference: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/ + language: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/language/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/language/ + language-functions: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/language/#functions + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/language/#functions + language-index: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/language/#functions + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/language/#functions + print-all-labels-from-a-classic-condition: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/examples/#print-all-labels-from-a-classic-condition + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/examples/#print-all-labels-from-a-classic-condition + template-annotations-and-labels: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/ +--- + +# Annotation and label template reference + +Annotations and labels in alert rules can be defined using plain text. However, you can also define templates to customize their values with dynamic data from alert rule queries. + +For example, you can template the `summary` annotation to include information from query values, providing relevant alert context for responders. Refer to [Template annotations and labels](ref:template-annotations-and-labels) for various use cases. + +In templates, variables represent dynamic values from queries, while functions perform actions to transform or format this data. + +## Variables + +Variables represent dynamic values from alert rule queries that can be displayed or accessed in your templates. + +The `$` and `.` symbols are used to reference variables and their properties. You can reference variables directly in your alert rule definitions using the `$` symbol followed by the variable name. Similarly, you can access properties of variables using the dot (`.`) notation in alert rule templates. + +``` +{{ $values.A.Value }} +``` + +Templates are based on the **Go templating system**. Refer to [Template language](ref:language) for additional information. + +The following variables are available when templating annotations and labels: + +| Variables | Description | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| [$labels](#labels) | Contains all labels from the query. | +| [$values](#values) | Contains the labels and floating point values of all instant queries and expressions, indexed by their Ref IDs. | +| [$value](#value) | A string containing the labels and values of all instant queries; threshold, reduce and math expressions, and classic conditions in the alert rule. | + +### $labels + +The `$labels` variable contains all labels from the query. + +{{< figure src="/media/docs/alerting/query-labels-and_value.png" max-width="1200px" caption="An alert rule displaying labels and value from a query" >}} + +For example, suppose you have a query that returns CPU usage for all of your servers, and you have an alert rule that fires when any of your servers have exceeded 80% CPU usage for the last 5 minutes. You want to add a summary annotation to the alert that tells you which server is experiencing high CPU usage. With the `$labels` variable you can write a template that prints a human-readable sentence such as: + +``` +CPU usage for {{ $labels.instance }} has exceeded 80% for the last 5 minutes +``` + +The outcome of this template would be: + +``` +CPU usage for server1 has exceeded 80% for the last 5 minutes +``` + +> If you are using a classic condition then `$labels` will not contain any labels from the query. Classic conditions discard these labels in order to enforce uni-dimensional behavior (at most one alert per alert rule). If you want to use labels from the query in your template then use the example [here](ref:print-all-labels-from-a-classic-condition). + +### $values + +The `$values` variable is a table containing the labels and floating point values of all instant queries and expressions, indexed by their Ref IDs (e.g, `A`, `B`, `C`, etc.). It does not contain the results of range queries, as they can return hundreds or thousands of rows. + +Each Ref IDs, such as `$values.A`, has the following properties + +| Property | Type | Description | +| -------- | --------------- | ------------------------------------------------------------ | +| `Value` | Float | The value returned by the instant query or expression. | +| `Labels` | Key/value pairs | The labels associated with the instance query or expression. | + +Here's the previous example printing now the value of the instant query with Ref ID `A`: + +``` +{{ $values.A.Value }} CPU usage for {{ $labels.instance }} over the last 5 minutes. +``` + +If the alert has the label `instance=server1` and the query returns `81.2345`, the template would print: + +``` +81.2345 CPU usage for instance1 over the last 5 minutes. +``` + +If the query in Ref ID `A` is a range query rather than an instant query then add a reduce expression with Ref ID `B` and replace `$values.A.Value` with `$values.B.Value`: + +``` +{{ $values.B.Value }} CPU usage for {{ $labels.instance }} over the last 5 minutes. +``` + +Alternatively, you can use the `index()` function to retrieve the query value: + +``` +{{ index $values "B" }} CPU usage for {{ index $labels "instance" }} over the last 5 minutes. +``` + +#### $value + +The `$value` variable is a string containing the labels and values of all instant queries; threshold, reduce and math expressions, and classic conditions in the alert rule. + +This example prints the `$value` variable: + +``` +{{ $value }}: CPU usage has exceeded 80% for the last 5 minutes. +``` + +It would display something like this: + +``` +[ var='A' labels={instance=instance1} value=81.234 ]: CPU usage has exceeded 80% for the last 5 minutes. +``` + +Instead, we recommend using [$values](#values), which contains the same information as `$value` but is structured in an easier-to-use table format. + +## Functions + +Functions can perform actions in templates such as transforming or formatting data. + +Note that the [functions provided by Go's template language](ref:language-functions), such as `index`, `and`, `printf`, and `len`, are available, along with many others. + +In addition, the following functions are also available for templating annotations and labels: + +**Numbers** + +| Name | Arguments | Returns | Description | +| ----------------------------------------- | ---------------- | ------- | ---------------------------------------------------------------- | +| [humanize](#humanize) | number or string | string | Humanizes decimal numbers. | +| [humanize1024](#humanize1024) | number or string | string | Like `humanize`, but but uses 1024 as the base rather than 1000. | +| [humanizeDuration](#humanizeduration) | number or string | string | Humanizes a duration in seconds. | +| [humanizePercentage](#humanizepercentage) | number or string | string | Humanizes a ratio value to a percentage. | +| [humanizeTimestamp](#humanizetimestamp) | number or string | string | Humanizes a Unix timestamp. | +| [toTime](#totime) | number or string | time | Converts a Unix timestamp in seconds to time. | + +**Strings** + +| Name | Arguments | Returns | Description | +| ------------------------------- | -------------------------- | ------- | --------------------------------------------------------------------------------------------- | +| [title](#title) | string | string | Capitalizes the first character of each word. | +| [toUpper](#toupper) | string | string | Returns all text in uppercase. | +| [toLower](#tolower) | string | string | Returns all text in lowercase. | +| [stripPort](#stripport) | string | string | Returns only host. | +| [match](#match) | pattern, text | boolean | Matches the text against a regular expression pattern. | +| [reReplaceAll](#rereplaceall) | pattern, replacement, text | string | Replaces text matching the regular expression. | +| [graphLink](#graphlink) | expr | string | Returns the path to the graphical view in `Explore` for the given expression and data source. | +| [tableLink](#tablelink) | expr | string | Returns the path to the tabular view in `Explore` for the given expression and data source. | +| [parseDuration](#parseduration) | string | float | Parses a duration string such as "1h" into the number of seconds it represents. | +| [stripDomain](#stripdomain) | string | string | Returns the result of removing the domain part of a FQDN. | + +**Others** + +| Name | Arguments | Returns | Description | +| --------------------------- | ------------- | ---------------------- | -------------------------------------------------------------------------------- | +| [args](#args) | []interface{} | map[string]interface{} | Translates a list of objects to a map with keys arg0, arg1 etc. | +| [safeHtml](#safehtml) | string | string | Marks string as HTML not requiring auto-escaping. | +| [externalURL](#externalurl) | none | string | Returns the external URL of the Grafana server as configured in the ini file(s). | +| [pathPrefix](#pathprefix) | none | string | Returns the path of the Grafana server as configured in the ini file(s). | + +For further context on these functions, note that templating in Grafana is based on the [Prometheus template implementation](https://prometheus.io/docs/prometheus/latest/configuration/template_reference/), enabling the use of these functions and Prometheus-like templates for formatting alert messages within Grafana. + +#### humanize + +The `humanize` function humanizes decimal numbers: + +``` +{{ humanize 1000.0 }} +``` + +``` +1k +``` + +#### humanize1024 + +The `humanize1024` works similar to `humanize` but but uses 1024 as the base rather than 1000: + +``` +{{ humanize1024 1024.0 }} +``` + +``` +1ki +``` + +#### humanizeDuration + +The `humanizeDuration` function humanizes a duration in seconds: + +``` +{{ humanizeDuration 60.0 }} +``` + +``` +1m 0s +``` + +#### humanizePercentage + +The `humanizePercentage` function humanizes a ratio value between 0 and 1 to a percentage: + +``` +{{ humanizePercentage 0.2 }} +``` + +``` +20% +``` + +#### humanizeTimestamp + +The `humanizeTimestamp` function humanizes a Unix timestamp: + +``` +{{ humanizeTimestamp 1577836800.0 }} +``` + +``` +2020-01-01 00:00:00 +0000 UTC +``` + +#### toTime + +The `toTime` function converts a Unix timestamp in seconds to time.: + +``` +{{ toTime 1727802106 }} +``` + +``` +2024-10-01 17:01:46 +0000 UTC +``` + +#### title + +The `title` function capitalizes the first character of each word: + +``` +{{ title "hello, world!" }} +``` + +``` +Hello, World! +``` + +#### toUpper + +The `toUpper` function returns all text in uppercase: + +``` +{{ toUpper "Hello, world!" }} +``` + +``` +HELLO, WORLD! +``` + +#### toLower + +The `toLower` function returns all text in lowercase: + +``` +{{ toLower "Hello, world!" }} +``` + +``` +hello, world! +``` + +#### stripPort + +The `stripPort` splits string into host and port, then returns only host: + +``` +{{ stripPort "example.com:8080" }} +``` + +``` +example.com +``` + +#### match + +The `match` function matches the text against a regular expression pattern: + +``` +{{ match "a.*" "abc" }} +``` + +``` +true +``` + +#### reReplaceAll + +The `reReplaceAll` function replaces text matching the regular expression: + +``` +{{ reReplaceAll "localhost:(.*)" "example.com:$1" "localhost:8080" }} +``` + +``` +example.com:8080 +``` + +#### graphLink + +The `graphLink` function returns the path to the graphical view in [Explore](ref:explore) for the given expression and data source: + +``` +{{ graphLink "{\"expr\": \"up\", \"datasource\": \"gdev-prometheus\"}" }} +``` + +``` +/explore?left=["now-1h","now","gdev-prometheus",{"datasource":"gdev-prometheus","expr":"up","instant":false,"range":true}] +``` + +#### parseDuration + +The `parseDuration` function parses a duration string such as "1h" into the number of seconds it represents. + +``` +{{ parseDuration "1h" }} +``` + +``` +3600 +``` + +#### stripDomain + +The `stripDomain` removes the domain part of a FQDN, leaving port untouched: + +``` +{{ stripDomain "example.com:8080" }} +``` + +``` +example:8080 +``` + +#### tableLink + +The `tableLink` function returns the path to the tabular view in [Explore](ref:explore) for the given expression and data source: + +``` +{{ tableLink "{\"expr\": \"up\", \"datasource\": \"gdev-prometheus\"}" }} +``` + +``` +/explore?left=["now-1h","now","gdev-prometheus",{"datasource":"gdev-prometheus","expr":"up","instant":true,"range":false}] +``` + +#### args + +The `args` function translates a list of objects to a map with keys arg0, arg1 etc. This is intended to allow multiple arguments to be passed to templates: + +``` +{{define "x"}}{{.arg0}} {{.arg1}}{{end}}{{template "x" (args 1 "2")}} +``` + +``` +1 2 +``` + +#### safeHtml + +The `safeHtml` function marks string as HTML not requiring auto-escaping: + +``` +{{ safeHtml "Text"}} +``` + +``` +Text +``` + +#### externalURL + +The `externalURL` function returns the external URL of the Grafana server as configured in the ini file(s): + +``` +{{ externalURL }} +``` + +``` +https://example.com/grafana +``` + +#### pathPrefix + +The `pathPrefix` function returns the path of the Grafana server as configured in the ini file(s): + +``` +{{ pathPrefix }} +``` + +``` +/grafana +``` + +## Differences with notification templates + +Both notification templates and alert rule templates use the Go templating system. However, the [functions and variables available in notification templates](ref:notification-template-reference) differ from those used in annotations and labels templates, which are described in this documentation. + +Annotation and label templates operate in the context of an individual alert instance, while notification templates apply to a notification that includes a group of alert(s). + +For example, notification templates provide the `.Alerts` variable, which includes the list of all firing and resolved alerts in the notification. This variable is not available in alert rule templates, which operate within the context of a single alert instance. + +Additionally, you cannot reuse templates for labels and annotations as you can with notification templates. Instead, you need to write each template inline within the label or annotation fields and manually copy them wherever you want to reuse the templates. diff --git a/docs/sources/alerting/alerting-rules/templating-labels-annotations.md b/docs/sources/alerting/alerting-rules/templating-labels-annotations.md deleted file mode 100644 index d25fa40daec..00000000000 --- a/docs/sources/alerting/alerting-rules/templating-labels-annotations.md +++ /dev/null @@ -1,459 +0,0 @@ ---- -aliases: - - ../fundamentals/annotation-label/variables-label-annotation/ # /docs/grafana//alerting/fundamentals/annotation-label/variables-label-annotation/ -canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templating-labels-annotations/ -description: Learn about how to template labels and annotations -keywords: - - grafana - - alerting - - templating - - labels - - annotations -labels: - products: - - cloud - - enterprise - - oss -title: Template labels and annotations -weight: 500 -refs: - explore: - - pattern: /docs/ - destination: /docs/grafana//explore/ ---- - -# Template labels and annotations - -You can use templates to include data from queries and expressions in labels and annotations. For example, you might want to set the severity label for an alert based on the value of the query, or use the instance label from the query in a summary annotation so you know which server is experiencing high CPU usage. - -When using custom labels with templates it is important to make sure that the label value does not change between consecutive evaluations of the alert rule as this will end up creating large numbers of distinct alerts. However, it is OK for the template to produce different label values for different alerts. For example, do not put the value of the query in a custom label as this will end up creating a new set of alerts each time the value changes. Instead use annotations. - -All templates should be written in [text/template](https://pkg.go.dev/text/template). Regardless of whether you are templating a label or an annotation, you should write each template inline inside the label or annotation that you are templating. This means you cannot share templates between labels and annotations, and instead you will need to copy templates wherever you want to use them. - -Each template is evaluated whenever the alert rule is evaluated, and is evaluated for every alert separately. For example, if your alert rule has a templated summary annotation, and the alert rule has 10 firing alerts, then the template will be executed 10 times, once for each alert. You should try to avoid doing expensive computations in your templates as much as possible. - -{{% admonition type="caution" %}} -Extra whitespace in label templates can break matches with notification policies. -{{% /admonition %}} - -## Variables - -In Grafana templating, the `$` and `.` symbols are used to reference variables and their properties. You can reference variables directly in your alert rule definitions using the `$` symbol followed by the variable name. Similarly, you can access properties of variables using the dot (`.`) notation within alert rule definitions. - -The following variables are available to you when templating labels and annotations: - -### The labels variable - -The `$labels` variable contains all labels from the query. For example, suppose you have a query that returns CPU usage for all of your servers, and you have an alert rule that fires when any of your servers have exceeded 80% CPU usage for the last 5 minutes. You want to add a summary annotation to the alert that tells you which server is experiencing high CPU usage. With the `$labels` variable you can write a template that prints a human-readable sentence such as: - -``` -CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes -``` - -> If you are using a classic condition then `$labels` will not contain any labels from the query. Classic conditions discard these labels in order to enforce uni-dimensional behavior (at most one alert per alert rule). If you want to use labels from the query in your template then use the example [here](#print-all-labels-from-a-classic-condition). - -### The value variable - -The `$value` variable is a string containing the labels and values of all instant queries; threshold, reduce and math expressions, and classic conditions in the alert rule. It does not contain the results of range queries, as these can return anywhere from 10s to 10,000s of rows or metrics. If it did, for especially large queries a single alert could use 10s of MBs of memory and Grafana would run out of memory very quickly. - -To print the `$value` variable in the summary you would write something like this: - -``` -CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes: {{ $value }} -``` - -And would look something like this: - -``` -CPU usage for instance1 has exceeded 80% for the last 5 minutes: [ var='A' labels={instance=instance1} value=81.234 ] -``` - -Here `var='A'` refers to the instant query with Ref ID A, `labels={instance=instance1}` refers to the labels, and `value=81.234` refers to the average CPU usage over the last 5 minutes. - -If you want to print just some of the string instead of the full string then use the `$values` variable. It contains the same information as `$value`, but in a structured table, and is much easier to use then writing a regular expression to match just the text you want. - -### The values variable - -The `$values` variable is a table containing the labels and floating point values of all instant queries and expressions, indexed by their Ref IDs. - -To print the value of the instant query with Ref ID A: - -``` -CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes: {{ index $values "A" }} -``` - -For example, given an alert with the labels `instance=server1` and an instant query with the value `81.2345`, this would print: - -``` -CPU usage for instance1 has exceeded 80% for the last 5 minutes: 81.2345 -``` - -If the query in Ref ID A is a range query rather than an instant query then add a reduce expression with Ref ID B and replace `(index $values "A")` with `(index $values "B")`: - -``` -CPU usage for {{ index $labels "instance" }} has exceeded 80% for the last 5 minutes: {{ index $values "B" }} -``` - -## Examples - -The following examples attempt to show the most common use-cases we have seen for templates. You can use these examples verbatim, or adapt them as necessary for your use case. For more information on how to write text/template refer see [the beginner's guide to alert notification templates in Grafana](https://grafana.com/blog/2023/04/05/grafana-alerting-a-beginners-guide-to-templating-alert-notifications/). - -### Print all labels, comma separated - -To print all labels, comma separated, print the `$labels` variable: - -``` -{{ $labels }} -``` - -For example, given an alert with the labels `alertname=High CPU usage`, `grafana_folder=CPU alerts` and `instance=server1`, this would print: - -``` -alertname=High CPU usage, grafana_folder=CPU alerts, instance=server1 -``` - -> If you are using classic conditions then `$labels` will not contain any labels from the query. Refer to [the $labels variable](#the-labels-variable) for more information. - -### Print all labels, one per line - -To print all labels, one per line, use a `range` to iterate over each key/value pair and print them individually. Here `$k` refers to the name and `$v` refers to the value of the current label: - -``` -{{ range $k, $v := $labels -}} -{{ $k }}={{ $v }} -{{ end }} -``` - -For example, given an alert with the labels `alertname=High CPU usage`, `grafana_folder=CPU alerts` and `instance=server1`, this would print: - -``` -alertname=High CPU usage -grafana_folder=CPU alerts -instance=server1 -``` - -> If you are using classic conditions then `$labels` will not contain any labels from the query. Refer to [the $labels variable](#the-labels-variable) for more information. - -### Print an individual label - -To print an individual label use the `index` function with the `$labels` variable: - -``` -The host {{ index $labels "instance" }} has exceeded 80% CPU usage for the last 5 minutes -``` - -For example, given an alert with the labels `instance=server1`, this would print: - -``` -The host server1 has exceeded 80% CPU usage for the last 5 minutes -``` - -> If you are using classic conditions then `$labels` will not contain any labels from the query. Refer to [the $labels variable](#the-labels-variable) for more information. - -### Print the value of a query - -To print the value of an instant query you can print its Ref ID using the `index` function and the `$values` variable: - -``` -{{ index $values "A" }} -``` - -For example, given an instant query that returns the value 81.2345, this will print: - -``` -81.2345 -``` - -To print the value of a range query you must first reduce it from a time series to an instant vector with a reduce expression. You can then print the result of the reduce expression by using its Ref ID instead. For example, if the reduce expression takes the average of A and has the Ref ID B you would write: - -``` -{{ index $values "B" }} -``` - -### Print the humanized value of a query - -To print the humanized value of an instant query use the `humanize` function: - -``` -{{ humanize (index $values "A").Value }} -``` - -For example, given an instant query that returns the value 81.2345, this will print: - -``` -81.234 -``` - -To print the humanized value of a range query you must first reduce it from a time series to an instant vector with a reduce expression. You can then print the result of the reduce expression by using its Ref ID instead. For example, if the reduce expression takes the average of A and has the Ref ID B you would write: - -``` -{{ humanize (index $values "B").Value }} -``` - -### Print the value of a query as a percentage - -To print the value of an instant query as a percentage use the `humanizePercentage` function: - -``` -{{ humanizePercentage (index $values "A").Value }} -``` - -This function expects the value to be a decimal number between 0 and 1. If the value is instead a decimal number between 0 and 100 you can either divide it by 100 in your query or using a math expression. If the query is a range query you must first reduce it from a time series to an instant vector with a reduce expression. - -### Set a severity from the value of a query - -To set a severity label from the value of a query use an if statement and the greater than comparison function. Make sure to use decimals (`80.0`, `50.0`, `0.0`, etc) when doing comparisons against `$values` as text/template does not support type coercion. You can find a list of all the supported comparison functions [here](https://pkg.go.dev/text/template#hdr-Functions). - -``` -{{ if (gt $values.A.Value 80.0) -}} -high -{{ else if (gt $values.A.Value 50.0) -}} -medium -{{ else -}} -low -{{- end }} -``` - -### Print all labels from a classic condition - -You cannot use `$labels` to print labels from the query if you are using classic conditions, and must use `$values` instead. The reason for this is classic conditions discard these labels to enforce uni-dimensional behavior (at most one alert per alert rule). If classic conditions didn't discard these labels, then queries that returned many time series would cause alerts to flap between firing and resolved constantly as the labels would change every time the alert rule was evaluated. - -Instead, the `$values` variable contains the reduced values of all time series for all conditions that are firing. For example, if you have an alert rule with a query A that returns two time series, and a classic condition B with two conditions, then `$values` would contain `B0`, `B1`, `B2` and `B3`. If the classic condition B had just one condition, then `$values` would contain just `B0` and `B1`. - -To print all labels of all firing time series use the following template (make sure to replace `B` in the regular expression with the Ref ID of the classic condition if it's different): - -``` -{{ range $k, $v := $values -}} -{{ if (match "B[0-9]+" $k) -}} -{{ $k }}: {{ $v.Labels }}{{ end }} -{{ end }} -``` - -For example, a classic condition for two time series exceeding a single condition would print: - -``` -B0: instance=server1 -B1: instance=server2 -``` - -If the classic condition has two or more conditions, and a time series exceeds multiple conditions at the same time, then its labels will be duplicated for each condition that is exceeded: - -``` -B0: instance=server1 -B1: instance=server2 -B2: instance=server1 -B3: instance=server2 -``` - -If you need to print unique labels you should consider changing your alert rules from uni-dimensional to multi-dimensional instead. You can do this by replacing your classic condition with reduce and math expressions. - -### Print all values from a classic condition - -To print all values from a classic condition take the previous example and replace `$v.Labels` with `$v.Value`: - -``` -{{ range $k, $v := $values -}} -{{ if (match "B[0-9]+" $k) -}} -{{ $k }}: {{ $v.Value }}{{ end }} -{{ end }} -``` - -For example, a classic condition for two time series exceeding a single condition would print: - -``` -B0: 81.2345 -B1: 84.5678 -``` - -If the classic condition has two or more conditions, and a time series exceeds multiple conditions at the same time, then `$values` will contain the values of all conditions: - -``` -B0: 81.2345 -B1: 92.3456 -B2: 84.5678 -B3: 95.6789 -``` - -## Functions - -The following functions are available to you when templating labels and annotations: - -### args - -The `args` function translates a list of objects to a map with keys arg0, arg1 etc. This is intended to allow multiple arguments to be passed to templates: - -``` -{{define "x"}}{{.arg0}} {{.arg1}}{{end}}{{template "x" (args 1 "2")}} -``` - -``` -1 2 -``` - -### externalURL - -The `externalURL` function returns the external URL of the Grafana server as configured in the ini file(s): - -``` -{{ externalURL }} -``` - -``` -https://example.com/grafana -``` - -### graphLink - -The `graphLink` function returns the path to the graphical view in [Explore](ref:explore) for the given expression and data source: - -``` -{{ graphLink "{\"expr\": \"up\", \"datasource\": \"gdev-prometheus\"}" }} -``` - -``` -/explore?left=["now-1h","now","gdev-prometheus",{"datasource":"gdev-prometheus","expr":"up","instant":false,"range":true}] -``` - -### humanize - -The `humanize` function humanizes decimal numbers: - -``` -{{ humanize 1000.0 }} -``` - -``` -1k -``` - -### humanize1024 - -The `humanize1024` works similar to `humanize` but but uses 1024 as the base rather than 1000: - -``` -{{ humanize1024 1024.0 }} -``` - -``` -1ki -``` - -### humanizeDuration - -The `humanizeDuration` function humanizes a duration in seconds: - -``` -{{ humanizeDuration 60.0 }} -``` - -``` -1m 0s -``` - -### humanizePercentage - -The `humanizePercentage` function humanizes a ratio value to a percentage: - -``` -{{ humanizePercentage 0.2 }} -``` - -``` -20% -``` - -### humanizeTimestamp - -The `humanizeTimestamp` function humanizes a Unix timestamp: - -``` -{{ humanizeTimestamp 1577836800.0 }} -``` - -``` -2020-01-01 00:00:00 +0000 UTC -``` - -### match - -The `match` function matches the text against a regular expression pattern: - -``` -{{ match "a.*" "abc" }} -``` - -``` -true -``` - -### pathPrefix - -The `pathPrefix` function returns the path of the Grafana server as configured in the ini file(s): - -``` -{{ pathPrefix }} -``` - -``` -/grafana -``` - -### tableLink - -The `tableLink` function returns the path to the tabular view in [Explore](ref:explore) for the given expression and data source: - -``` -{{ tableLink "{\"expr\": \"up\", \"datasource\": \"gdev-prometheus\"}" }} -``` - -``` -/explore?left=["now-1h","now","gdev-prometheus",{"datasource":"gdev-prometheus","expr":"up","instant":true,"range":false}] -``` - -### title - -The `title` function capitalizes the first character of each word: - -``` -{{ title "hello, world!" }} -``` - -``` -Hello, World! -``` - -### toLower - -The `toLower` function returns all text in lowercase: - -``` -{{ toLower "Hello, world!" }} -``` - -``` -hello, world! -``` - -### toUpper - -The `toUpper` function returns all text in uppercase: - -``` -{{ toUpper "Hello, world!" }} -``` - -``` -HELLO, WORLD! -``` - -### reReplaceAll - -The `reReplaceAll` function replaces text matching the regular expression: - -``` -{{ reReplaceAll "localhost:(.*)" "example.com:$1" "localhost:8080" }} -``` - -``` -example.com:8080 -``` diff --git a/docs/sources/alerting/configure-notifications/_index.md b/docs/sources/alerting/configure-notifications/_index.md index 9e334326cb5..71acdff8fc4 100644 --- a/docs/sources/alerting/configure-notifications/_index.md +++ b/docs/sources/alerting/configure-notifications/_index.md @@ -26,9 +26,9 @@ refs: destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/notification-policies/ templates-page: - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/fundamentals/notifications/templates/ + destination: /docs/grafana//alerting/fundamentals/templates/ - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/templates/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/templates/ --- # Configure notifications diff --git a/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/configure-email.md b/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/configure-email.md index 3b2b1898dc5..a8aacba6623 100644 --- a/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/configure-email.md +++ b/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/configure-email.md @@ -14,12 +14,20 @@ labels: menuTitle: Email title: Configure email for Alerting weight: 0 +refs: + notification-templates: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/ --- # Configure email for Alerting Use the Grafana Alerting - email integration to send email notifications when your alerts are firing. An email is sent when an alert fires and when an alert gets resolved. +Note that you can customize the `subject` and `message` of the email using [notification templates](ref:notification-templates). However, you cannot add HTML and CSS to email notifications for visual changes. + ## Before you begin {{}} diff --git a/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/configure-slack.md b/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/configure-slack.md index 359cebc1bd3..e5821985bf9 100644 --- a/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/configure-slack.md +++ b/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/configure-slack.md @@ -15,6 +15,11 @@ menuTitle: Slack title: Configure Slack for Alerting weight: 0 refs: + notification-templates: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/ nested-policy: - pattern: /docs/grafana/ destination: /docs/grafana//alerting/configure-notifications/create-notification-policy/#add-new-nested-policy @@ -36,9 +41,9 @@ There are two ways of integrating Slack into Grafana Alerting. Webhooks is the simpler way to post messages into Slack. Slack automatically creates a bot user with all the necessary permissions to post messages to one particular channel of your choice. -{{< admonition type="note" >}} -Grafana Alerting only allows one Slack channel per contact point. -{{< /admonition >}} +Note that you can only setup one Slack channel per contact point. + +You can customize the `title` and `body` of the Slack message using [notification templates](ref:notification-templates); however, you cannot modify its visual appearance with custom blocks. ## Before you begin @@ -53,17 +58,12 @@ If you are using a Slack API Token, complete the following steps. 1. Right click the channel you want to receive notifications in. 1. Click View channel details. 1. Scroll down and copy the Channel ID. - {{< admonition type="note" >}} - While going through these steps, Slack may prompt you to Reinstall your app in order for the changes to take effect. - {{< /admonition >}} + + Note that while going through these steps, Slack may prompt you to Reinstall your app in order for the changes to take effect. ### Webhook URL -If you are using a Webhook URL, follow steps 1 and 5 in the [Slack API Quickstart](https://api.slack.com/start/quickstart). - -{{< admonition type="note" >}} -Make sure you copy the Slack app Webhook URL. You need this when setting up your contact point integration in Grafana Alerting. -{{< /admonition >}} +If you are using a Webhook URL, follow steps 1 and 5 in the [Slack API Quickstart](https://api.slack.com/start/quickstart), and copy the Slack app Webhook URL. You need this when setting up your contact point integration in Grafana Alerting. ## Procedure diff --git a/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/configure-teams.md b/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/configure-teams.md index 4f720bb616d..b788f21c5ff 100644 --- a/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/configure-teams.md +++ b/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/configure-teams.md @@ -14,12 +14,20 @@ labels: menuTitle: Microsoft Teams title: Configure Microsoft Teams for Alerting weight: 0 +refs: + notification-templates: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/ --- # Configure Microsoft Teams for Alerting Use the Grafana Alerting - Microsoft Teams integration to receive notifications in your team’s channel when your alerts are firing. +Note that you can customize the `title` and `message` of the notification using [notification templates](ref:notification-templates); however, you cannot modify its visual appearance with adaptive cards. + ## Before you begin To set up Microsoft Teams for integration with Grafana Alerting, create a new workflow that accepts Webhook requests. This allows Grafana to send alert notifications to Microsoft Teams channels. diff --git a/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier.md b/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier.md index a1329970152..f08319ecd50 100644 --- a/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier.md +++ b/docs/sources/alerting/configure-notifications/manage-contact-points/integrations/webhook-notifier.md @@ -22,6 +22,12 @@ labels: menuTitle: Webhook notifier title: Configure the webhook notifier for Alerting weight: 0 +refs: + notification-templates: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/ --- # Configure the webhook notifier for Alerting @@ -139,7 +145,11 @@ The webhook notification is a simple way to send information about a state chang | imageURL | string | URL of a screenshot of a panel assigned to the rule that created this notification | {{< admonition type="note" >}} -Alert rules are not coupled to dashboards anymore therefore the fields related to dashboards `dashboardId` and `panelId` have been removed. + +You can customize the `title` and `message` fields using [notification templates](ref:notification-templates). + +However, you cannot customize webhook data structure or format, including JSON fields or sending data in XML, nor can you change the webhook HTTP headers. + {{< /admonition >}} ## Procedure diff --git a/docs/sources/alerting/configure-notifications/template-notifications/_index.md b/docs/sources/alerting/configure-notifications/template-notifications/_index.md index a3947af0eaa..744fa22c4a2 100644 --- a/docs/sources/alerting/configure-notifications/template-notifications/_index.md +++ b/docs/sources/alerting/configure-notifications/template-notifications/_index.md @@ -13,55 +13,98 @@ labels: - cloud - enterprise - oss -title: Configure notification messages +title: Template notifications weight: 430 refs: - use-notification-templates: + template-annotations-and-labels: - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/configure-notifications/template-notifications/use-notification-templates/ + destination: /docs/grafana//alerting/alerting-rules/templates/ - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/use-notification-templates/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/ + manage-notification-templates: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/manage-notification-templates/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/manage-notification-templates/ reference: - pattern: /docs/grafana/ destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/ - pattern: /docs/grafana-cloud/ destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/ - using-go-templating-language: + examples: - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/configure-notifications/template-notifications/using-go-templating-language/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/examples/ - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/using-go-templating-language/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/examples/ --- -# Configure notification messages +# Template notifications -Customize the content of your notifications with notifications templates. +You can use notification templates to change the title, message, and format of notifications. -You can use notification templates to change the title, message, and format of the message in your notifications. +Grafana provides a **default template** for notification titles (`default.title`) and one default template for notification messages (`default.message`). Both templates display common alert details. -Notification templates are not tied to specific contact point integrations, such as email or Slack. However, you can choose to create separate notification templates for different contact point integrations. +You can also create a notification template to customize the content and format of your notification messages. For example: -You can use notification templates to: - -- Customize content: Personalize the subject of an email or the title of a message. Modify text within notifications, like selecting or omitting certain labels, annotations, and links. Format text with bold and italic styles, and add or remove line breaks. +- Personalize the subject of an email or the title of a message. +- Modify text within notifications, like selecting or omitting certain labels, annotations, and links. +- Format text with bold and italic styles, and add or remove line breaks. However, there are limitations. You cannot: -- Modify Visual Appearance: Add HTML and CSS to email notifications for visual changes. Alter the design of notifications in messaging services like Slack and Microsoft Teams, such as adding custom blocks or adaptive cards. -- Manage Media and Data: Adjust the number and size of images or their placement in notifications. Customize webhook data structure or format, including JSON fields or sending data in XML. Modify HTTP headers in webhooks beyond those in the contact point configuration. +- Modify Visual Appearance: Add HTML or CSS to email notifications for visual changes. Alter the design of notifications in messaging services like Slack or Microsoft Teams, such as adding custom blocks or adaptive cards. +- Manage Media and Data: Customize the data structure or format passed to the templates, like adding new JSON fields or sending XML data for webhooks. Modify HTTP headers in webhooks beyond those defined in the configuration, or adjust the number, size, or placement of images. -## Learn more +Here's an [example](ref:examples) that displays the summary and description annotations for each alert in the notification: -[Using Go's templating language](ref:using-go-templating-language) +```go +{{ define "custom.alerts" -}} +{{ len .Alerts }} alert(s) +{{ range .Alerts -}} + {{ template "alert.summary_and_description" . -}} +{{ end -}} +{{ end -}} +{{ define "alert.summary_and_description" }} + Summary: {{.Annotations.summary}} + Status: {{ .Status }} + Description: {{.Annotations.description}} +{{ end -}} +``` -Learn how to write the content of your notification templates in Go’s templating language. +The notification message would look like this: -Create reusable notification templates for your contact points. +``` +2 alert(s) -[Use notification templates](ref:use-notification-templates) + Summary: The database server db1 has exceeded 75% of available disk space. + Status: firing + Description: This alert fires when a database server is at risk of running out of disk space. You should take measures to increase the maximum available disk space as soon as possible to avoid possible corruption. -Use notification templates to send notifications to your contact points. + Summary: The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes. + Status: resolved + Description: This alert fires when a web server responds with more 5xx errors than is expected. This could be an issue with the web server or a backend service. +``` -[Reference](ref:reference) +{{% admonition type="note" %}} +Avoid adding extra information about alert instances in notification templates, as this information will only be visible in the notification message. -Data that is available when writing templates. +Instead, you should [use annotations or labels](ref:template-annotations-and-labels) to add information directly to the alert, ensuring it's also visible in the alert state and alert history within Grafana. You can then print the new alert annotation or label in notification templates. +{{% /admonition %}} + +#### Select a notification template for a contact point + +Notification templates are not tied to specific contact point integrations, such as email or Slack, and the same template can be shared across multiple contact points. + +The notification template is assigned to the contact point to determine the notification message sent to contact point integrations. + +{{< figure src="/media/docs/alerting/how-notification-templates-works.png" max-width="1200px" caption="A flow of the alert notification process, from querying the alert rule to sending the alert notification message." >}} + +By default, Grafana provides default templates, such as `default.title` and `default.message`, to format notification messages. + +## More information + +For further details on how to write notification templates, refer to: + +- [Select, create, and preview a notification template](ref:manage-notification-templates) +- [Notification template reference](ref:reference) +- [Notification template examples](ref:examples) diff --git a/docs/sources/alerting/configure-notifications/template-notifications/create-notification-templates.md b/docs/sources/alerting/configure-notifications/template-notifications/create-notification-templates.md deleted file mode 100644 index 0e6d8a78eb3..00000000000 --- a/docs/sources/alerting/configure-notifications/template-notifications/create-notification-templates.md +++ /dev/null @@ -1,376 +0,0 @@ ---- -aliases: - - ../../manage-notifications/template-notifications/create-notification-templates/ # /docs/grafana//alerting/manage-notifications/template-notifications/create-notification-templates/ -canonical: https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/create-notification-templates/ -description: Create notification templates to sent to your contact points -keywords: - - grafana - - alerting - - notifications - - templates - - create templates - - edit templates - - delete templates -labels: - products: - - cloud - - enterprise - - oss -title: Create notification templates -weight: 200 ---- - -# Create notification templates - -Create reusable notification templates to send to your contact points. - -You can add one or more templates to your notification template. - -Your notification template name must be unique. You cannot have two templates with the same name in the same notification template or in different notification templates. Avoid defining templates with the same name as default templates, such as: `__subject`, `__text_values_list`, `__text_alert_list`, `default.title` and `default.message`. - -To create a notification template, complete the following steps. - -1. Click **Alerts & IRM** -> **Contact points**. -1. Click the **Notification Templates** tab and then **+ Add notification template**. - -1. Enter a name for the notification template. - -1. Write the content of the template in the content field. - -1. Save your changes. - -`{{ define "email.subject" }}` and `{{ end }}` is automatically added to the start and end of the content: - -To create a notification template that contains more than one template: - -1. Click **+ Add notification template**. - -2. Enter a name for the notification template. - -3. Write each template in the Content field, including `{{ define "name-of-template" }}` and `{{ end }}` at the start and end of each template. - -4. Save your changes. - -## Preview notification templates - -Preview how your notification templates should look before using them in your contact points, helping you understand the result of the template you are creating as well as enabling you to fix any errors before saving it. - -**Note:** This feature is only for Grafana Alertmanager. - -To preview your notification templates: - -1. Navigate to **Alerts&IRM** -> **Alerting** -> **Contact points** -> **Notification Templates**. -1. Click **+ Add notification template** or edit an existing template. -1. Add or update your template content. - - Default data is provided and you can add or edit alert data to it as well as alert instances. You can add alert data directly in the Payload data window itself or click **Select alert instances** or **Add custom alerts**. - -1. Optional: To add alert data from existing alert instances: - - a. Click **Select alert instances**. - - b. Hover over the alert instances to view more information on each alert instance. - - c. Click **Confirm** to add the alert instance(s) to the payload. - -1. Optional: To add alert data using the Alert data editor, click **Add custom data:** - - a. Add annotations, custom labels and/or set a dashboard or a panel. - - b. Toggle Firing/resolved depending on whether you want to add firing or resolved alerts to your notification. - - c. Click **Add alert data**. - - d. Click **Refresh preview** to see what your template content should look like and the corresponding payload data. - - If there are any errors in your template, they are displayed in the Preview and you can correct them before saving. - -1. Save your changes. - -## Template the subject of an email - -Template the subject of an email to contain the number of firing and resolved alerts: - -``` -1 firing alert(s), 0 resolved alerts(s) -``` - -1. Create a template called `email.subject` with the following content: - - ``` - {{ define "email.subject" }} - {{ len .Alerts.Firing }} firing alert(s), {{ len .Alerts.Resolved }} resolved alert(s) - {{ end }} - ``` - -2. Execute the template from the subject field in your contact point integration: - - ``` - {{ template "email.subject" . }} - ``` - -## Template the message of an email - -Template the message of an email to contain a summary of all firing and resolved alerts: - -``` -There are 2 firing alert(s), and 1 resolved alert(s) - -Firing alerts: - -- alertname=Test 1 grafana_folder=GrafanaCloud has value(s) B=1 -- alertname=Test 2 grafana_folder=GrafanaCloud has value(s) B=2 - -Resolved alerts: - -- alertname=Test 3 grafana_folder=GrafanaCloud has value(s) B=0 -``` - -1. Create a notification template called `email` with two templates in the content: `email.message_alert` and `email.message`. - - The `email.message_alert` template is used to print the labels and values for each firing and resolved alert while the `email.message` template contains the structure of the email. - - ``` - {{- define "email.message_alert" -}} - {{- range .Labels.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }} has value(s) - {{- range $k, $v := .Values }} {{ $k }}={{ $v }}{{ end }} - {{- end -}} - - {{ define "email.message" }} - There are {{ len .Alerts.Firing }} firing alert(s), and {{ len .Alerts.Resolved }} resolved alert(s) - - {{ if .Alerts.Firing -}} - Firing alerts: - {{- range .Alerts.Firing }} - - {{ template "email.message_alert" . }} - {{- end }} - {{- end }} - - {{ if .Alerts.Resolved -}} - Resolved alerts: - {{- range .Alerts.Resolved }} - - {{ template "email.message_alert" . }} - {{- end }} - {{- end }} - - {{ end }} - ``` - -2. Execute the template from the message field in your contact point integration: - - ``` - {{ template "email.message" . }} - ``` - -## Group multiple alert instances into one email notification - -To make alerts more concise, you can group multiple instances of a firing alert into a single email notification in a table format. This way, you avoid long, repetitive emails and make alerts easier to digest. - -Follow these steps to create a custom notification template that consolidates alert instances into a table. - -1. Modify the alert rule to include an annotation that is referenced in the notification template later on. -1. Enter a name for the **custom annotation**: In this example, _ServerInfo_. -1. Enter the following code as the value for the annotation. It retrieves the server's instance name and a corresponding metric value, formatted as a table row: - - ``` - {{ index $labels "instance" }}{{- "\t" -}}{{ index $values "A"}}{{- "\n" -}} - ``` - - This line of code returns the labels and their values in the form of a table. Assuming $labels has `{"instance": "node1"}` and $values has `{"A": "123"}`, the output would be: - - ``` - node1 123 - ``` - -1. Create a notification template that references the _ServerInfo_ annotation. - - ```go - {{ define "Table" }} - {{- "\nHost\t\tValue\n" -}} - {{ range .Alerts -}} - {{ range .Annotations.SortedPairs -}} - {{ if (eq .Name "ServerInfo") -}} - {{ .Value -}} - {{- end }} - {{- end }} - {{- end }} - {{ end }} - ``` - - The notification template outputs a list of server information from the "ServerInfo" annotation for each alert instance. - -1. Navigate to your contact point in Grafana -1. In the **Message** field, reference the template by name (see **Optional Email settings** section): - - ``` - {{ template "Table" . }} - ``` - - This generates a neatly formatted table in the email, grouping information for all affected servers into a single notification. - -## Conditional notification template - -Template alert notifications based on a label. In this example the label represents a namespace. - -1. Use the following code in your notification template to display different messages based on the namespace: - - ```go - {{ define "my_conditional_notification" }} - {{ if eq .CommonLabels.namespace "namespace-a" }} - Alert: CPU limits have reached 80% in namespace-a. - {{ else if eq .CommonLabels.namespace "namespace-b" }} - Alert: CPU limits have reached 80% in namespace-b. - {{ else if eq .CommonLabels.namespace "namespace-c" }} - Alert: CPU limits have reached 80% in namespace-c. - {{ else }} - Alert: CPU limits have reached 80% for {{ .CommonLabels.namespace }} namespace. - {{ end }} - {{ end }} - ``` - - `.CommonLabels` is a map containing the labels that are common to all the alerts firing. - - Make sure to replace the `.namespace` label with a label that exists in your alert rule. - -1. Replace `namespace-a`, `namespace-b`, and `namespace-c` with your specific namespace values. -1. Navigate to your contact point in Grafana -1. In the **Message** field, reference the template by name (see **Optional settings** section): - - ``` - {{ template "my_conditional_notification" . }} - ``` - - This template alters the content of alert notifications depending on the namespace value. - -## Template the title of a Slack message - -Template the title of a Slack message to contain the number of firing and resolved alerts: - -``` -1 firing alert(s), 0 resolved alerts(s) -``` - -1. Create a template called `slack.title` with the following content: - - ``` - {{ define "slack.title" }} - {{ len .Alerts.Firing }} firing alert(s), {{ len .Alerts.Resolved }} resolved alert(s) - {{ end }} - ``` - -2. Execute the template from the title field in your contact point integration: - - ``` - {{ template "slack.title" . }} - ``` - -## Template the content of a Slack message - -Template the content of a Slack message to contain a description of all firing and resolved alerts, including their labels, annotations, Silence URL and Dashboard URL. - -**Note:** - -This template is for Grafana-managed alerts only. -To use the template for Grafana Mimir/Loki-managed alerts, delete the references to DashboardURL and SilenceURL. -For more information, see the [Prometheus documentation on notifications](https://prometheus.io/docs/alerting/latest/notifications/). - -``` -1 firing alert(s): - -[firing] Test1 -Labels: -- alertname: Test1 -- grafana_folder: GrafanaCloud -Annotations: -- description: This is a test alert -Silence: https://example.com/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DTest1&matcher=grafana_folder%3DGrafanaCloud -Go to dashboard: https://example.com/d/dlhdLqF4z?orgId=1 - -1 resolved alert(s): - -[firing] Test2 -Labels: -- alertname: Test2 -- grafana_folder: GrafanaCloud -Annotations: -- description: This is another test alert -Silence: https://example.com/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DTest2&matcher=grafana_folder%3DGrafanaCloud -Go to dashboard: https://example.com/d/dlhdLqF4z?orgId=1 -``` - -1. Create a template called `slack` with two templates in the content: `slack.print_alert` and `slack.message`. - - The `slack.print_alert` template is used to print the labels, annotations, SilenceURL and DashboardURL while the `slack.message` template contains the structure of the notification. - - ``` - {{ define "slack.print_alert" -}} - [{{.Status}}] {{ .Labels.alertname }} - Labels: - {{ range .Labels.SortedPairs -}} - - {{ .Name }}: {{ .Value }} - {{ end -}} - {{ if .Annotations -}} - Annotations: - {{ range .Annotations.SortedPairs -}} - - {{ .Name }}: {{ .Value }} - {{ end -}} - {{ end -}} - {{ if .SilenceURL -}} - Silence: {{ .SilenceURL }} - {{ end -}} - {{ if .DashboardURL -}} - Go to dashboard: {{ .DashboardURL }} - {{- end }} - {{- end }} - - {{ define "slack.message" -}} - {{ if .Alerts.Firing -}} - {{ len .Alerts.Firing }} firing alert(s): - {{ range .Alerts.Firing }} - {{ template "slack.print_alert" . }} - {{ end -}} - {{ end }} - {{ if .Alerts.Resolved -}} - {{ len .Alerts.Resolved }} resolved alert(s): - {{ range .Alerts.Resolved }} - {{ template "slack.print_alert" .}} - {{ end -}} - {{ end }} - {{- end }} - ``` - -2. Execute the template from the text body field in your contact point integration: - - ``` - {{ template "slack.message" . }} - ``` - -## Template both email and Slack with shared templates - -Instead of creating separate notification templates for email and Slack, you can share the same template. - -For example, if you want to send an email with this subject and Slack message with this title: - -``` -1 firing alert(s), 0 resolved alerts(s) -``` - -1. Create a template called `common.subject_title` with the following content: - - ``` - {{ define "common.subject_title" }} - {{ len .Alerts.Firing }} firing alert(s), {{ len .Alerts.Resolved }} resolved alert(s) - {{ end }} - ``` - -2. For email, execute the template from the subject field in your email contact point integration: - - ``` - {{ template "common.subject_title" . }} - ``` - -3. For Slack, execute the template from the title field in your Slack contact point integration: - - ``` - {{ template "common.subject_title" . }} - ``` diff --git a/docs/sources/alerting/configure-notifications/template-notifications/examples.md b/docs/sources/alerting/configure-notifications/template-notifications/examples.md new file mode 100644 index 00000000000..939880eef58 --- /dev/null +++ b/docs/sources/alerting/configure-notifications/template-notifications/examples.md @@ -0,0 +1,323 @@ +--- +canonical: https://grafana.com/docs/grafana/latest/alerting/alerting-rules/templates/examples/ +description: Examples of notification templates +keywords: + - grafana + - alerting + - templating + - notification templates +labels: + products: + - cloud + - enterprise + - oss +title: Notification template examples +menuTitle: Examples +weight: 103 +refs: + template-annotations-and-labels: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/ + template-notifications: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/ + manage-notification-templates: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/manage-notification-templates/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/manage-notification-templates/ + reference: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/ + reference-notification-data: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/#notification-data + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/#notification-data + reference-alert: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/#alert + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/#alert + language: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/language/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/language/ +--- + +# Notification template examples + +Notification templates allows you to change the default notification messages. + +You can modify the content and format of notification messages. For example, you can customize the content to show only specific information or adjust the format to suit a particular contact point, such as Slack or Email. + +{{% admonition type="note" %}} +Avoid adding extra information about alert instances in notification templates, as this information is only be visible in the notification message. + +Instead, you should [use annotations or labels](ref:template-annotations-and-labels) to add information directly to the alert, ensuring it's also visible in the alert state and alert history within Grafana. You can then print the new alert annotation or label in notification templates. +{{% /admonition %}} + +This page provides various examples illustrating how to template common notification messages. For more details about notification templates, refer to: + +- [Template notifications](ref:template-notifications) +- [Select, create, and preview a notification template](ref:manage-notification-templates) +- [Notification template reference](ref:reference) + +## Basic examples + +Notification templates can access the [notification data](ref:reference-notification-data) using the dot (`.`). The following examples demonstrate some basic uses of the [template language](ref:language). + +For instance, to check if there are common labels (`.CommonLabels`) for all alerts in the notification, use `if`: + +```go +{{ define "custom_message" -}} +{{ if .CommonLabels }} +Alerts have common labels +{{ else }} +There are no common labels +{{ end }} +{{ end }} +``` + +To iterate on the alerts in the notification and print a specific label, use `range` and `index`: + +```go +{{ define "custom_message" -}} +{{ range .Alerts }} +The name of the alert is {{ index .Labels "alertname" }} +{{ end }} +{{ end }} +``` + +Alternatively, you can use the `.` notation to print the value of the key. + +```go +{{ define "custom_message" -}} +{{ range .Alerts }} +The name of the alert is {{ .Labels.alertname }} +{{ end }} +{{ end }} +``` + +```template_output +The name of the alert is InstanceDown + +The name of the alert is CpuOverload +``` + +## Print alerts with summary and description + +Here's an example that displays the summary and description annotations for each alert in the notification. + +```go +{{ define "custom.alerts" -}} +{{ len .Alerts }} alert(s) +{{ range .Alerts -}} + {{ template "alert.summary_and_description" . -}} +{{ end -}} +{{ end -}} +{{ define "alert.summary_and_description" }} + Summary: {{.Annotations.summary}} + Status: {{ .Status }} + Description: {{.Annotations.description}} +{{ end -}} +``` + +In this example: + +- A template (`alert.summary_and_description`) is defined to print the `summary`, `status`, and `description` of one [alert](ref:reference-alert). +- The main template `custom.alerts` iterates the list of alerts (`.Alerts`) in [notification data](ref:reference-notification-data), executing the `alert.summary_and_description` template to print the details of each alert. + +The notification message would look like this: + +```template_output +2 alert(s) + + Summary: The database server db1 has exceeded 75% of available disk space. + Status: firing + Description: This alert fires when a database server is at risk of running out of disk space. You should take measures to increase the maximum available disk space as soon as possible to avoid possible corruption. + + Summary: The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes. + Status: resolved + Description: This alert fires when a web server responds with more 5xx errors than is expected. This could be an issue with the web server or a backend service. +``` + +## Print firing and resolved alerts + +The following example is similar to the previous one, but it separates firing and resolved alerts. + +```go +{{ define "custom.firing_and_resolved_alerts" -}} +{{ len .Alerts.Resolved }} resolved alert(s) +{{ range .Alerts.Resolved -}} + {{ template "alert.summary_and_description" . -}} +{{ end }} +{{ len .Alerts.Firing }} firing alert(s) +{{ range .Alerts.Firing -}} + {{ template "alert.summary_and_description" . -}} +{{ end -}} +{{ end -}} +{{ define "alert.summary_and_description" }} + Summary: {{.Annotations.summary}} + Status: {{ .Status }} + Description: {{.Annotations.description}} +{{ end -}} +``` + +Instead of `.Alerts`, the template accesses `.Alerts.Firing` and `.Alerts.Resolved` separately to print details for each alert. + +The output might now look like this: + +```template_output +1 resolved alert(s) + + Summary: The database server db1 has exceeded 75% of available disk space. + Status: resolved + Description: This alert fires when a database server is at risk of running out of disk space. You should take measures to increase the maximum available disk space as soon as possible to avoid possible corruption. + +1 firing alert(s) + + Summary: The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes. + Status: firing + Description: This alert fires when a web server responds with more 5xx errors than is expected. This could be an issue with the web server or a backend service. +``` + +## Print common labels and annotations + +This example displays only the labels and annotations that are common to all alerts in the notification. + +```go +{{ define "custom.common_labels_and_annotations" -}} +{{ len .Alerts.Resolved }} resolved alert(s) +{{ len .Alerts.Firing }} firing alert(s) +Common labels: {{ len .CommonLabels.SortedPairs }} +{{ range .CommonLabels.SortedPairs -}} +- {{ .Name }} = {{ .Value }} +{{ end }} +Common annotations: {{ len .CommonAnnotations.SortedPairs }} +{{ range .CommonAnnotations.SortedPairs }} +- {{ .Name }} = {{ .Value }} +{{ end }} +{{ end -}} +``` + +Note that `.CommonAnnotations` and `.CommonLabels` are part of [notification data](ref:reference-notification-data). + +```template_output +1 resolved alert(s) +1 firing alert(s) +Common labels: 2 +- grafana_folder = server_alerts +- team = server_admin + +Common annotations: 0 +``` + +## Print individual labels and annotations + +This example displays all labels and annotations for each [alert](ref:reference-alert) in the notification. + +```go +{{ define "custom.alert_labels_and_annotations" -}} +{{ len .Alerts.Resolved }} resolved alert(s) +{{ range .Alerts.Resolved -}} + {{ template "alert.labels_and_annotations" . -}} +{{ end }} +{{ len .Alerts.Firing }} firing alert(s) +{{ range .Alerts.Firing -}} + {{ template "alert.labels_and_annotations" . -}} +{{ end -}} +{{ end -}} +{{ define "alert.labels_and_annotations" }} +Alert labels: {{ len .Labels.SortedPairs }} +{{ range .Labels.SortedPairs -}} +- {{ .Name }} = {{ .Value }} +{{ end -}} +Alert annotations: {{ len .Annotations.SortedPairs }} +{{ range .Annotations.SortedPairs -}} +- {{ .Name }} = {{ .Value }} +{{ end -}} +{{ end -}} +``` + +In this example: + +- The `custom.alert_labels_and_annotations` template iterates over the list of resolved and firing alerts, similar to previous examples. It then executes `alert.labels_and_annotations` for each alert. +- The `alert.labels_and_annotations` template prints all the alert labels and annotations by accessing `.Labels.SortedPairs` and `.Annotations.SortedPairs`. + +```template_output +1 resolved alert(s) + +Alert labels: 4 +- alertname = db_server_disk_space +- grafana_folder = server_alerts +- server = db1 +- team = server_admin + +Alert annotations: 2 +- summary = The database server db1 has exceeded 75% of available disk space. +- description = This alert fires when a database server is at risk of running out of disk space. You should take measures to increase the maximum available disk space as soon as possible to avoid possible corruption. + +1 firing alert(s) + +Alert labels: 4 +- alertname = web_server_http_errors +- grafana_folder = server_alerts +- server = web1 +- team = server_admin + +Alert annotations: 2 +- summary = The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes. +- description = This alert fires when a web server responds with more 5xx errors than is expected. This could be an issue with the web server or a backend service. +``` + +## Print URLs for runbook and alert data in Grafana + +Note that the following example works only for Grafana-managed alerts. It displays some [alert data](ref:reference-alert) such as `DashboardURL`, `PanelURL`, and `SilenceURL`, which are exclusive to Grafana-managed alerts. + +```go +{{ define "custom.alert_additional_details" -}} +{{ len .Alerts.Resolved }} resolved alert(s) +{{ range .Alerts.Resolved -}} + {{ template "alert.additional_details" . -}} +{{ end }} +{{ len .Alerts.Firing }} firing alert(s) +{{ range .Alerts.Firing -}} + {{ template "alert.additional_details" . -}} +{{ end -}} +{{ end -}} +{{ define "alert.additional_details" }} +- Dashboard: {{ .DashboardURL }} +- Panel: {{ .PanelURL }} +- AlertGenerator: {{ .GeneratorURL }} +- Silence: {{ .SilenceURL }} +- RunbookURL: {{ .Annotations.runbook_url}} +{{ end -}} +``` + +The output of this template looks like this: + +```template_output +1 resolved alert(s) + +- Dashboard: https://example.com/d/ +- Panel: https://example.com/d/ +- AlertGenerator: ?orgId=1 +- Silence: https://example.com/alerting/silence/new +- RunbookURL: https://example.com/on-call/db_server_disk_space + +1 firing alert(s) + +- Dashboard: https://example.com/d/ +- Panel: https://example.com/d/ +- AlertGenerator: ?orgId=1 +- Silence: https://example.com/alerting/silence/new +- RunbookURL: https://example.com/on-call/web_server_http_errors +``` diff --git a/docs/sources/alerting/configure-notifications/template-notifications/images-in-notifications.md b/docs/sources/alerting/configure-notifications/template-notifications/images-in-notifications.md index 5b9324f6fc6..57484aaa6ed 100644 --- a/docs/sources/alerting/configure-notifications/template-notifications/images-in-notifications.md +++ b/docs/sources/alerting/configure-notifications/template-notifications/images-in-notifications.md @@ -14,7 +14,8 @@ labels: - enterprise - oss title: Use images in notifications -weight: 500 +menuTitle: Use images +weight: 105 --- # Use images in notifications @@ -55,6 +56,8 @@ Refer to the table at the end of this page for a list of contact points and thei 8. Grafana does not delete screenshots uploaded to its internal web server. To delete screenshots from `static_root_path/images/attachments` after a certain amount of time, we recommend setting up a CRON job. +9. Note you cannot adjust the number and size of images or their placement in notifications. + ## Configuration {{% admonition type="note" %}} @@ -130,7 +133,7 @@ If Grafana has been set up to send images in notifications, however notification 5. If images cannot be taken because of mis-configuration or an issue with image rendering there are logs for `Failed to take an image` including the Dashboard UID, Panel ID, and the error message. 6. Check that the contact point supports images in notifications and whether it supports uploading images to the receiving service or referencing images that have been uploaded to a cloud storage service. -## Metrics +## Monitor Grafana provides the following metrics to observe the performance and failure rate of images in notifications. For example, if a screenshot could not be taken within the expected time (10 seconds) then the counter `grafana_screenshot_failures_total` is updated. diff --git a/docs/sources/alerting/configure-notifications/template-notifications/language.md b/docs/sources/alerting/configure-notifications/template-notifications/language.md new file mode 100644 index 00000000000..9f47303daff --- /dev/null +++ b/docs/sources/alerting/configure-notifications/template-notifications/language.md @@ -0,0 +1,85 @@ +--- +aliases: + - ../../manage-notifications/template-notifications/using-go-templating-language/ # /docs/grafana//alerting/manage-notifications/template-notifications/using-go-templating-language/ + - ../../configure-notifications/template-notifications/using-go-templating-language/ # /docs/grafana//alerting/configure-notifications/template-notifications/using-go-templating-language/ +weight: 104 +canonical: https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/language/ +description: Use Go template language to create your notification and alert rule templates +keywords: + - grafana + - alerting + - templates + - write templates +labels: + products: + - cloud + - enterprise + - oss +title: Alerting template language +menuTitle: Template language +refs: + alert-rule-template-reference: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/reference/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/ + alert-rule-template-reference-variables: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/reference/#variables + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/#variables + notification-template-reference: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/ + reference-notificationdata: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/#notification-data + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/#notification-data +--- + +# Alerting template language + +Notification templates and alert rule templates, such as annotations and labels, both use the Go template language, [text/template](https://pkg.go.dev/text/template). + +Both types of templates can use the same keywords, functions, and comparison operators of the Go template language, such as `range`, `if`, `and`, `index`, `eq`, and more. + +However, it's important to note that because notifications and alert rules operate in distinct context, some additional variables and functions are only available for either notification or alert rule templates. Refer to: + +- [Annotation and label template reference](ref:alert-rule-template-reference) +- [Notification template reference](ref:notification-template-reference) + +This documentation provides an overview of the functions and operators of the Go template language that are available for both notification and alert rule templates. + +## Print + +To print the value of something, use `{{` and `}}`. You can print the value of a [variable](#variables), a field of a variable, the result of a function, or the value of dot. + +``` +{{ $values }} +{{ $values.A.Value }} +{{ humanize 1000.0 }} +{{ .Alerts }} +``` + +## Dot + +In `text/template`, there is a special cursor called dot, written as `.`. You can think of this cursor as a variable whose value changes depending on where in the template it is used. + +At the start of notification templates, dot (`.`) refers to [Notification Data](ref:reference-notificationdata). + +``` +{{ .Alerts }} +``` + +In annotation and label templates, dot (`.`) is initialized with all alert data. It’s recommended to use the [`$labels` and `$values` variables](ref:alert-rule-template-reference-variables) instead to directly access the alert labels and query values. + +{{% admonition type="note" %}} +Dot (`.`) might refer to something else when used in a [range](#range), a [with](#with), or when writing [templates](#templates) used in other templates. +{{% /admonition %}} + +[//]: <> (The above section is not included in the shared file because `refs` links are not supported in shared files.) + +{{< docs/shared lookup="alerts/template-language.md" source="grafana" version="" >}} diff --git a/docs/sources/alerting/configure-notifications/template-notifications/manage-notification-templates.md b/docs/sources/alerting/configure-notifications/template-notifications/manage-notification-templates.md new file mode 100644 index 00000000000..5ebeb4882c0 --- /dev/null +++ b/docs/sources/alerting/configure-notifications/template-notifications/manage-notification-templates.md @@ -0,0 +1,136 @@ +--- +aliases: + - ../../manage-notifications/template-notifications/create-notification-templates/ # /docs/grafana//alerting/manage-notifications/template-notifications/create-notification-templates/ + - ../../manage-notifications/template-notifications/use-notification-templates/ # /docs/grafana//alerting/manage-notifications/template-notifications/use-notification-templates/ + - ../../configure-notifications/template-notifications/use-notification-templates/ # /docs/grafana//alerting/manage-notifications/template-notifications/use-notification-templates/ + - ../../configure-notifications/template-notifications/create-notification-templates/ # /docs/grafana//alerting/manage-notifications/template-notifications/create-notification-templates/ +canonical: https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/manage-notification-templates/ +description: Create notification templates to sent to your contact points +keywords: + - grafana + - alerting + - notifications + - templates + - create templates + - edit templates + - delete templates +labels: + products: + - cloud + - enterprise + - oss +title: Manage notification templates +menuTitle: Manage templates +weight: 101 +refs: + notification-template-reference: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/ + notification-template-examples: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/language/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/language/ +--- + +# Manage notification templates + +In contact points, you can select notification templates to customize the notification messages sent. + +By default, Grafana provides a template for the notification title (`default.title`) and a template for the notification message (`default.message`). Both default templates display common alert details. + +You can also create custom templates to customize the content and format of notification messages, which can then be applied to one or more contact points. + +This documentation provides step-by-step instructions for selecting templates in contact points, previewing templates, and creating custom templates using the Grafana UI. + +## Select a notification template for a contact point + +To add an existing notification template to your contact point, complete the following steps. + +1. Click an existing contact point or create a new one. +1. In **Optional settings**, click any field that contains templates. + + For example, if you are creating an email contact point integration, click **Message** or **Subject**. + +1. Click **Edit**. + A dialog box opens where you can select templates. +1. Click **Select existing template** to select a template and [preview](#preview-a-notification-template) it using the default payload. + + You can also copy the selected template and use it in the custom tab. + +1. Click **Enter custom message** to customize and edit the field directly. Note that the title changes depending on the field you are editing. + +1. You can switch between the two tabs to access the list of available templates and copy them across to the customized version. + +1. Click **Save contact point**. + +## Create a notification template + +Create notification templates to customize notification messages and reuse them in contact points. + +Your notification template name must be unique. You cannot have two templates with the same name in the same notification template or in different notification templates. Avoid defining templates with the same name as default templates, such as: `__subject`, `__text_values_list`, `__text_alert_list`, `default.title` and `default.message`. + +To create a notification template in Grafana, complete the following steps. + +1. Click **Alerts & IRM** -> **Contact points**. +1. Click the **Notification Templates** tab and then **+ Add notification template**. + +1. Enter a name for the notification template. + +1. Write the content of the template in the content field. + +1. Save your changes. + + If `{{ define }}` is not included in the content, `{{ define "" }}` and `{{ end }}` is automatically added to the start and end. + +To create a notification template that contains more than one template, complete the following steps. + +1. Click **+ Add notification template**. + +1. Enter a name for the notification template. + +1. Write each template in the Content field, including `{{ define "name-of-template" }}` and `{{ end }}` at the start and end of each template. + +1. Save your changes. + +For more details on how to write notification templates, refer to the [template reference](ref:notification-template-reference) and [examples](ref:notification-template-examples). + +## Preview a notification template + +Preview how your notification templates should look before using them in your contact points, helping you understand the result of the template you are creating as well as enabling you to fix any errors before saving it. + +{{% admonition type="note" %}} +Notification template preview is only for Grafana Alertmanager. +{{% /admonition %}} + +To preview your notification templates: + +1. Navigate to **Alerts&IRM** -> **Alerting** -> **Contact points** -> **Notification Templates**. +1. Click **+ Add notification template** or edit an existing template. +1. Add or update your template content. + + Default data is provided and you can add or edit alert data to it as well as alert instances. You can add alert data directly in the Payload data window itself or click **Select alert instances** or **Add custom alerts**. + +1. Optional: To add alert data from existing alert instances: + + a. Click **Select alert instances**. + + b. Hover over the alert instances to view more information on each alert instance. + + c. Click **Confirm** to add the alert instance(s) to the payload. + +1. Optional: To add alert data using the Alert data editor, click **Add custom data:** + + a. Add annotations, custom labels and/or set a dashboard or a panel. + + b. Toggle Firing/resolved depending on whether you want to add firing or resolved alerts to your notification. + + c. Click **Add alert data**. + + d. Click **Refresh preview** to see what your template content should look like and the corresponding payload data. + + If there are any errors in your template, they are displayed in the Preview and you can correct them before saving. + +1. Save your changes. diff --git a/docs/sources/alerting/configure-notifications/template-notifications/reference.md b/docs/sources/alerting/configure-notifications/template-notifications/reference.md index f0b84f67d16..eafa9a02e29 100644 --- a/docs/sources/alerting/configure-notifications/template-notifications/reference.md +++ b/docs/sources/alerting/configure-notifications/template-notifications/reference.md @@ -13,80 +13,218 @@ labels: - cloud - enterprise - oss -title: Reference -weight: 400 +title: Notification template reference +menuTitle: Template reference +weight: 102 +refs: + alert-rule-template-reference: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/templates/reference/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/ + alert-grouping: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/fundamentals/notifications/group-alert-notifications/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/group-alert-notifications/ + template-language: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/language/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/language/ + template-language-functions: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/configure-notifications/template-notifications/language/#functions + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/language/#functions --- -# Reference +# Notification template reference -## Data +By default, Grafana provides predefined templates to format notification messages. -### Alert +You can also customize your notifications with custom templates, which are based on the [Go template language](ref:template-language). -| Name | Kind | Description | Example | -| ------------ | -------- | ----------------------------------------------------------------------------------- | --------------------- | -| Status | `string` | Firing or resolved | `{{ .Status }}` | -| Labels | `KV` | The labels for this alert | `{{ .Labels }}` | -| Annotations | `KV` | The annotations for this alert | `{{ .Annotations }}` | -| Values | `KV` | The values of all expressions, including Classic Conditions | `{{ .Values }}` | -| StartsAt | `Time` | The time the alert fired | `{{ .StartsAt }}` | -| EndsAt | `Time` | | `{{ .EndsAt }}` | -| GeneratorURL | `string` | A link to Grafana, or the source of the alert if using an external alert generator | `{{ .GeneratorURL }}` | -| SilenceURL | `string` | A link to silence the alert | `{{ .SilenceURL }}` | -| DashboardURL | `string` | A link to the Grafana Dashboard if the alert has a Dashboard UID annotation | `{{ .DashboardURL }}` | -| PanelURL | `string` | A link to the panel if the alert has a Panel ID annotation | `{{ .PanelURL }}` | -| Fingerprint | `string` | A unique string that identifies the alert | `{{ .Fingerprint }}` | -| ValueString | `string` | A string that contains the labels and value of each reduced expression in the alert | `{{ .ValueString }}` | +This documentation lists the data available for use in notification templates. -### ExtendedData +## Notification Data -| Name | Kind | Description | Example | -| ----------------- | --------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | -| Receiver | `string` | The name of the contact point sending the notification | `{{ .Receiver }}` | -| Status | `string` | The status is `firing` if at least one alert is firing, otherwise `resolved` | `{{ .Status }}` | -| Alerts | `[]Alert` | List of all firing and resolved alerts in this notification | `There are {{ len .Alerts }} alerts` | -| Firing alerts | `[]Alert` | List of all firing alerts in this notification | `There are {{ len .Alerts.Firing }} firing alerts` | -| Resolved alerts | `[]Alert` | List of all resolved alerts in this notification | `There are {{ len .Alerts.Resolved }} resolved alerts` | -| GroupLabels | `KV` | The labels that group these alerts in this | `{{ .GroupLabels }}` | -| CommonLabels | `KV` | The labels common to all alerts in this notification | `{{ .CommonLabels }}` | -| CommonAnnotations | `KV` | The annotations common to all alerts in this notification | `{{ .CommonAnnotations }}` | -| ExternalURL | `string` | A link to Grafana, or the Alertmanager that sent this notification if using an external Alertmanager | `{{ .ExternalURL }}` | +In notification templates, dot (`.`) is initialized with the following data: -### KV +| Name | Type | Description | +| ------------------- | ----------------- | ----------------------------------------------------------------------------------------------------- | +| `Receiver` | string | The name of the contact point sending the notification | +| `Status` | string | The status is `firing` if at least one alert is firing, otherwise `resolved`. | +| `Alerts` | [][Alert](#alert) | List of all firing and resolved alerts in this notification. | +| `Alerts.Firing` | [][Alert](#alert) | List of all firing alerts in this notification. | +| `Alerts.Resolved` | [][Alert](#alert) | List of all resolved alerts in this notification. | +| `GroupLabels` | [KV](#kv) | The labels that group these alerts in this notification based on the `Group by` option. | +| `CommonLabels` | [KV](#kv) | The labels common to all alerts in this notification. | +| `CommonAnnotations` | [KV](#kv) | The annotations common to all alerts in this notification. | +| `ExternalURL` | string | A link to Grafana, or the Alertmanager that sent this notification if using an external Alertmanager. | -`KV` is a set of key value pairs, where each key and value is a string. If a KV happens to contain numbers or bools then these are string representations of the numeric or boolean value. +It's important to remember that [a single notification can group multiple alerts](ref:alert-grouping) to reduce the number of alerts you receive. `Alerts` is an array that includes all the alerts in the notification. -Here is an example of a KV, the annotations of an alert: +Here's an example that prints all available notification data from dot (`.`): -```yaml -summary: 'A summary of the alert' -description: 'A description of the alert' +```go +{{ define "custom_template" }} + {{ .Receiver }} + {{ .Status }} + There are {{ len .Alerts }} alerts + There are {{ len .Alerts.Firing }} firing alerts + There are {{ len .Alerts.Resolved }} resolved alerts + {{ .GroupLabels }} + {{ .CommonLabels }} + {{ .CommonAnnotations }} + {{ .ExternalURL }} +{{ end }} ``` -In addition to iterating over each key value pair, you can sort the pairs, remove keys, and iterate over just the keys or the values. +## Alert -| Name | Description | Arguments | Returns | Example | -| ----------- | ---------------------------------------------- | --------- | ------- | ------------------------------------- | -| SortedPairs | Sorts | | | `{{ .Annotations.SortedPairs }}` | -| Remove | Returns a copy of the KV with the keys removed | []string | | `{{ .Annotations.Remove "summary" }}` | -| Names | A list of the names | | | `{{ .Names }}` | -| Values | A list of the values | | | `{{ .Values }}` | +`Alert` contains data for an individual alert: + +| Name | Type | Description | +| -------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| `Status` | string | Firing or resolved. | +| `Labels` | [KV](#kv) | The labels for this alert. | +| `Annotations` | [KV](#kv) | The annotations for this alert. | +| `StartsAt` | time | The time the alert fired | +| `EndsAt` | time | Only set if the end time of an alert is known. Otherwise set to a configurable timeout period from the time since the last alert was received. | +| `GeneratorURL` | string | A link to Grafana, or the source of the alert if using an external alert generator. | +| `Fingerprint` | string | A unique string that identifies the alert. | + +Grafana-managed alerts include these additional properties: + +| Name | Type | Description | +| -------------- | --------- | ------------------------------------------------------------------------------------ | +| `DashboardURL` | string | A link to the Grafana Dashboard if the alert has a Dashboard UID annotation. | +| `PanelURL` | string | A link to the panel if the alert has a Panel ID annotation. | +| `SilenceURL` | string | A link to silence the alert. | +| `Values` | [KV](#kv) | The values of all expressions, including Classic Conditions. | +| `ValueString` | string | A string that contains the labels and value of each reduced expression in the alert. | + +This example iterates over the list of firing and resolved alerts (`.Alerts`) in the notification and prints the data for each alert: + +```go +{{ define "custom_template" }} +{{ range .Alerts }} + {{ .Status }} + {{ .Labels }} + {{ .Annotations }} + {{ .StartsAt }} + {{ .EndsAt }} + {{ .GeneratorURL }} + {{ .Fingerprint }} + + {{/* Only available for Grafana-managed alerts */}} + {{ .DashboardURL }} + {{ .PanelURL }} + {{ .SilenceURL }} + {{ .Values }} + {{ .ValueString }} +{{ end }} +{{ end }} +``` + +## KV + +`KV` is a set of key value pairs, where each key and value is a string. + +Similarly to accessing variable properties, you can use `.` to retrieve the value of a value. For example: + +```go +{{ define "custom_template" }} + {{ .CommonLabels.grafana_folder }} +{{ end }} +``` + +If a KV happens to contain numbers or bools then these are string representations of the numeric or boolean value. + +Additionally, KV provides methods to sort the pairs, remove keys, and iterate over just the keys or values: + +| Method name | Description | Arguments | Returns | +| ----------- | ---------------------------------------------- | --------- | --------- | +| SortedPairs | Returns a sorted list of key/value pairs. | | Pairs | +| Remove | Returns a copy of the KV with the keys removed | []string | [KV](#kv) | +| Names | Return the names of the label names | | []string | +| Values | Return a list of the values | | []string | + +Here's an example of using these methods: + +```go +{{ define "custom_template" }} + {{ .CommonLabels.SortedPairs }} + {{ .CommonLabels.Names }} + {{ .CommonLabels.Values }} + {{ .CommonLabels.Remove (stringSlice "grafana_folder") }} +{{ end }} +``` + +## Functions + +Functions can perform actions in templates such as transforming or formatting data. + +Note that the [functions provided by Go's template language](ref:template-language-functions), such as `index`, `and`, `printf`, and `len`, are available, along with many others. + +In addition, the following functions are also available for templating notifications: + +### Strings + +| Name | Arguments | Returns | Description | +| -------------- | -------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------- | +| `title` | string | string | Capitalizes the first character of each word. | +| `toUpper` | string | string | Returns all text in uppercase. | +| `toLower` | string | string | Returns all text in lowercase. | +| `trimSpace` | string | string | Removes leading and trailing white spaces. | +| `match` | pattern, text | boolean | Matches the text against a regular expression pattern. | +| `reReplaceAll` | pattern, replacement, text | string | Replaces text matching the regular expression. | +| `join` | sep string, s []string | string | Concatenates the elements of s to create a single string. The separator string sep is placed between elements in the resulting string. | +| `safeHtml` | string | string | Marks string as HTML not requiring auto-escaping. | +| `stringSlice` | ...string | string | Returns the passed strings as a slice of strings. auto-escaping. | + +Here's an example of using these functions: + +```go +{{ define "custom_template" }} + {{ title "hello, world!" }} + {{ toUpper "Hello, world!" }} + {{ toLower "Hello, world!" }} + {{ trimSpace "Hello, world!" }} + {{ match "a.*" "abc" }} + {{ reReplaceAll "localhost:(.*)" "example.com:$1" "localhost:8080" }} + {{ join "-" (stringSlice "a" "b" "c") }} + {{ safeHtml "Text"}} + {{ stringSlice "a" "b" "c" }} +{{ end }} +``` ### Time -Time is from the Go [`time`](https://pkg.go.dev/time#Time) package. - You can format a time in a number of different formats using the `date` function. For example, to print the time that an alert fired in the format `15:04:05 MST`: -``` -{{ .StartsAt | date "15:04:05 MST" }} +```go +{{ define "custom_template" }} + {{ with (index .Alerts 0) }} + {{ .StartsAt | date "15:04:05 MST" }} + {{ end}} +{{ end }} ``` You can also use the `tz` function to change the timezone from UTC to a local time. For example: -``` +```go {{ .StartsAt | tz "Europe/Paris" | date "15:04:05 MST" }} ``` You can find a reference for Go's time format [here](https://pkg.go.dev/time#pkg-constants). + +## Differences with annotation and label templates + +In the alert rule, you can also template annotations and labels to include additional information. For example, you might add a `summary` annotation that displays the query value triggering the alert. + +Annotation and label templates add relevant information to individual alert instances, while notification templates inform about a group of alert instances. + +Since both types of templates operate in distinct contexts, the [functions and variables available in annotation and label templates](ref:alert-rule-template-reference) differ from those used in notification templates. diff --git a/docs/sources/alerting/configure-notifications/template-notifications/use-notification-templates.md b/docs/sources/alerting/configure-notifications/template-notifications/use-notification-templates.md deleted file mode 100644 index 3763e22ccf7..00000000000 --- a/docs/sources/alerting/configure-notifications/template-notifications/use-notification-templates.md +++ /dev/null @@ -1,53 +0,0 @@ ---- -aliases: - - ../../manage-notifications/template-notifications/use-notification-templates/ # /docs/grafana//alerting/manage-notifications/template-notifications/use-notification-templates/ -canonical: https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/use-notification-templates/ -description: Use notification templates in contact points to customize your notifications -keywords: - - grafana - - alerting - - notifications - - templates - - use templates -labels: - products: - - cloud - - enterprise - - oss -title: Use notification templates -weight: 300 -refs: - create-notification-templates: - - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/configure-notifications/template-notifications/create-notification-templates/ - - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/create-notification-templates/ - using-go-templating-language: - - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/configure-notifications/template-notifications/using-go-templating-language/ - - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/using-go-templating-language/ ---- - -# Use notification templates - -Use templates in contact points to customize your notifications. - -Complete the following steps to add templates to your contact point. - -1. Click an existing contact point or create a new one -1. In **Optional settings**, click any field that contains templates. - - For example, if you are creating an email contact point integration, click **Message** or **Subject**. - -1. Click **Edit**. - A dialog box opens where you can select templates. -1. Click **Select existing template** to select a template and preview it using the default payload. - - You can also copy the selected template and use it in the custom tab. - -1. Click **Enter custom message** to customize and edit the field directly. Note that the title changes depending on the field you are editing. - -1. You can switch between the two tabs to access the list of available templates and copy them across to the customized version. - -1. Click **Save contact point**. diff --git a/docs/sources/alerting/configure-notifications/template-notifications/using-go-templating-language.md b/docs/sources/alerting/configure-notifications/template-notifications/using-go-templating-language.md deleted file mode 100644 index 5999a20001e..00000000000 --- a/docs/sources/alerting/configure-notifications/template-notifications/using-go-templating-language.md +++ /dev/null @@ -1,298 +0,0 @@ ---- -aliases: - - ../../manage-notifications/template-notifications/using-go-templating-language/ # /docs/grafana//alerting/manage-notifications/template-notifications/using-go-templating-language/ -canonical: https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/using-go-templating-language/ -description: Use Go's templating language to create your own notification templates -keywords: - - grafana - - alerting - - notifications - - templates - - write templates -labels: - products: - - cloud - - enterprise - - oss -title: Using Go's templating language -weight: 100 -refs: - create-notification-templates: - - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/configure-notifications/template-notifications/create-notification-templates/ - - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/create-notification-templates/ - extendeddata: - - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/#extendeddata - - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/#extendeddata - reference: - - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/ - - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/ ---- - -# Using Go's templating language - -You write notification templates in Go's templating language, [text/template](https://pkg.go.dev/text/template). - -Before you start creating your own notification templates, we recommend that you read through this topic, which provides you with an overview of Go's templating language and writing templates in text/template. - -## Dot - -In text/template there is a special cursor called dot, and is written as `.`. You can think of this cursor as a variable whose value changes depending where in the template it is used. For example, at the start of a notification template `.` refers to something called [`ExtendedData`](ref:extendeddata) which contains a number of fields including `Alerts`, `Status`, `GroupLabels`, `CommonLabels`, `CommonAnnotations` and `ExternalURL`. However, dot might refer to something else when used in a range over a list, when used inside a `with`, or when writing feature templates to be used in other templates. You can see examples of this in [Create notification templates](ref:create-notification-templates), and all data and functions in the [Reference](ref:reference). - -## Opening and closing tags - -In text/template, templates start with `{{` and end with `}}` irrespective of whether the template prints a variable or executes control structures such as if statements. This is different from other templating languages such as Jinja where printing a variable uses `{{` and `}}` and control structures use `{%` and `%}`. - -## Print - -To print the value of something use `{{` and `}}`. You can print the value of dot, a field of dot, the result of a function, and the value of a [variable](#variables). For example, to print the `Alerts` field where dot refers to `ExtendedData` you would write the following: - -``` -{{ .Alerts }} -``` - -## Iterate over alerts - -To print just the labels of each alert, rather than all information about the alert, you can use a `range` to iterate the alerts in `ExtendedData`: - -``` -{{ range .Alerts }} -{{ .Labels }} -{{ end }} -``` - -Inside the range dot no longer refers to `ExtendedData`, but to an `Alert`. You can use `{{ .Labels }}` to print the labels of each alert. This works because `{{ range .Alerts }}` changes dot to refer to the current alert in the list of alerts. When the range is finished dot is reset to the value it had before the start of the range, which in this example is `ExtendedData`: - -``` -{{ range .Alerts }} -{{ .Labels }} -{{ end }} -{{/* does not work, .Labels does not exist here */}} -{{ .Labels }} -{{/* works, cursor was reset */}} -{{ .Status }} -``` - -## Iterate over annotations and labels - -Let's write a template to print the labels of each alert in the format `The name of the label is $name, and the value is $value`, where `$name` and `$value` contain the name and value of each label. - -Like in the previous example, use a range to iterate over the alerts in `.Alerts` such that dot refers to the current alert in the list of alerts, and then use a second range on the sorted labels so dot is updated a second time to refer to the current label. Inside the second range use `.Name` and `.Value` to print the name and value of each label: - -``` -{{ range .Alerts }} -{{ range .Labels.SortedPairs }} -The name of the label is {{ .Name }}, and the value is {{ .Value }} -{{ end }} -{{ range .Annotations.SortedPairs }} -The name of the annotation is {{ .Name }}, and the value is {{ .Value }} -{{ end }} -{{ end }} -``` - -## The index function - -To print a specific annotation or label use the `index` function. - -``` -{{ range .Alerts }} -The name of the alert is {{ index .Labels "alertname" }} -{{ end }} -``` - -## If statements - -You can use if statements in templates. For example, to print `There are no alerts` if there are no alerts in `.Alerts` you would write the following: - -``` -{{ if .Alerts }} -There are alerts -{{ else }} -There are no alerts -{{ end }} -``` - -## With - -With is similar to if statements, however unlike if statements, `with` updates dot to refer to the value of the with: - -``` -{{ with .Alerts }} -There are {{ len . }} alert(s) -{{ else }} -There are no alerts -{{ end }} -``` - -## Variables - -Variables in text/template must be created within the template. For example, to create a variable called `$variable` with the current value of dot you would write the following: - -``` -{{ $variable := . }} -``` - -You can use `$variable` inside a range or `with` and it will refer to the value of dot at the time the variable was defined, not the current value of dot. - -For example, you cannot write a template that use `{{ .Labels }}` in the second range because here dot refers to the current label, not the current alert: - -``` -{{ range .Alerts }} -{{ range .Labels.SortedPairs }} -{{ .Name }} = {{ .Value }} -{{/* does not work because in the second range . is a label not an alert */}} -There are {{ len .Labels }} -{{ end }} -{{ end }} -``` - -You can fix this by defining a variable called `$alert` in the first range and before the second range: - -``` -{{ range .Alerts }} -{{ $alert := . }} -{{ range .Labels.SortedPairs }} -{{ .Name }} = {{ .Value }} -{{/* works because $alert refers to the value of dot inside the first range */}} -There are {{ len $alert.Labels }} -{{ end }} -{{ end }} -``` - -## Range with index - -You can get the index of each alert within a range by defining index and value variables at the start of the range: - -``` -{{ $num_alerts := len .Alerts }} -{{ range $index, $alert := .Alerts }} -This is alert {{ $index }} out of {{ $num_alerts }} -{{ end }} -``` - -## Define templates - -You can define templates using `define` and the name of the template in double quotes. You should not define templates with the same name as other templates, including default templates such as `__subject`, `__text_values_list`, `__text_alert_list`, `default.title` and `default.message`. Where a template has been created with the same name as a default template, or a template in another notification template, Grafana might use either template. Grafana does not prevent, or show an error message, when there are two or more templates with the same name. - -``` -{{ define "print_labels" }} -{{ end }} -``` - -## Execute templates - -You can execute defined templates using `template`, the name of the template in double quotes, and the cursor that should be passed to the template: - -``` -{{ template "print_labels" . }} -``` - -## Pass data to templates - -Within a template dot refers to the value that is passed to the template. - -For example, if a template is passed a list of firing alerts then dot refers to that list of firing alerts: - -``` -{{ template "print_alerts" .Alerts }} -``` - -If the template is passed the sorted labels for an alert then dot refers to the list of sorted labels: - -``` -{{ template "print_labels" .SortedLabels }} -``` - -This is useful when writing reusable templates. For example, to print all alerts you might write the following: - -``` -{{ template "print_alerts" .Alerts }} -``` - -Then to print just the firing alerts you could write this: - -``` -{{ template "print_alerts" .Alerts.Firing }} -``` - -This works because both `.Alerts` and `.Alerts.Firing` is a list of alerts. - -``` -{{ define "print_alerts" }} -{{ range . }} -{{ template "print_labels" .SortedLabels }} -{{ end }} -{{ end }} -``` - -## Comments - -You can add comments with `{{/*` and `*/}}`: - -``` -{{/* This is a comment */}} -``` - -To prevent comments from adding line breaks use: - -``` -{{- /* This is a comment with no leading or trailing line breaks */ -}} -``` - -## Indentation - -You can use indentation, both tabs and spaces, and line breaks, to make templates more readable: - -``` -{{ range .Alerts }} - {{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }} - {{ end }} -{{ end }} -``` - -However, indentation in the template is also present in the text. Next, we look at how to remove it. - -## Remove spaces and line breaks - -In text/template use `{{-` and `-}}` to remove leading and trailing spaces and line breaks. - -For example, when using indentation and line breaks to make a template more readable: - -``` -{{ range .Alerts }} - {{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }} - {{ end }} -{{ end }} -``` - -The indentation and line breaks will also be present in the text: - -``` - alertname = "Test" - - grafana_folder = "Test alerts" -``` - -You can remove the indentation and line breaks from the text changing `}}` to `-}}` at the start of each range: - -``` -{{ range .Alerts -}} - {{ range .Labels.SortedPairs -}} - {{ .Name }} = {{ .Value }} - {{ end }} -{{ end }} -``` - -The indentation and line breaks in the template are now absent from the text: - -``` -alertname = "Test" -grafana_folder = "Test alerts" -``` diff --git a/docs/sources/alerting/fundamentals/alert-rules/annotation-label.md b/docs/sources/alerting/fundamentals/alert-rules/annotation-label.md index 50a8e74a82d..a39657fd63f 100644 --- a/docs/sources/alerting/fundamentals/alert-rules/annotation-label.md +++ b/docs/sources/alerting/fundamentals/alert-rules/annotation-label.md @@ -33,9 +33,9 @@ refs: destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-alerts-panels/ templates: - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/fundamentals/notifications/templates/ + destination: /docs/grafana//alerting/fundamentals/templates/ - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/templates/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/templates/ alert-rule-evaluation: - pattern: /docs/grafana/ destination: /docs/grafana//alerting/fundamentals/alert-rule-evaluation/ diff --git a/docs/sources/alerting/fundamentals/notifications/_index.md b/docs/sources/alerting/fundamentals/notifications/_index.md index 2017bfef8d8..aac1d6514f9 100644 --- a/docs/sources/alerting/fundamentals/notifications/_index.md +++ b/docs/sources/alerting/fundamentals/notifications/_index.md @@ -27,9 +27,9 @@ refs: destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/group-alert-notifications/ templates: - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/fundamentals/notifications/templates/ + destination: /docs/grafana//alerting/fundamentals/templates/ - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/templates/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/templates/ configure-alertmanager: - pattern: /docs/grafana/ destination: /docs/grafana//alerting/set-up/configure-alertmanager/ diff --git a/docs/sources/alerting/fundamentals/templates.md b/docs/sources/alerting/fundamentals/templates.md index 42b0900ff4c..8dabe94816c 100644 --- a/docs/sources/alerting/fundamentals/templates.md +++ b/docs/sources/alerting/fundamentals/templates.md @@ -1,10 +1,10 @@ --- aliases: - - ../fundamentals/notifications/templates/ # /docs/grafana//alerting/fundamentals/notifications/templates/ + - ../fundamentals/notifications/templates/ # /docs/grafana//alerting/fundamentals/templates/ - ../contact-points/message-templating/ # /docs/grafana//alerting/contact-points/message-templating/ - ../alert-rules/message-templating/ # /docs/grafana//alerting/alert-rules/message-templating/ - ../unified-alerting/message-templating/ # /docs/grafana//alerting/unified-alerting/message-templating/ -canonical: https://grafana.com/docs/grafana/latest/alerting/fundamentals/notifications/templates/ +canonical: https://grafana.com/docs/grafana/latest/alerting/fundamentals/templates/ description: Use templating to customize, format, and reuse alert notification messages. Create more flexible and informative alert notification messages by incorporating dynamic content, such as metric values, labels, and other contextual information. keywords: - grafana @@ -33,24 +33,19 @@ refs: destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/#annotations templating-labels-annotations: - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/alerting-rules/templating-labels-annotations/ + destination: /docs/grafana//alerting/alerting-rules/templates/ - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templating-labels-annotations/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/ notification-message-reference: - pattern: /docs/grafana/ destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/ - pattern: /docs/grafana-cloud/ destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/reference/ - notification-messages: + template-notifications: - pattern: /docs/grafana/ destination: /docs/grafana//alerting/configure-notifications/template-notifications/ - pattern: /docs/grafana-cloud/ destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/ - create-notification-templates: - - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/configure-notifications/template-notifications/create-notification-templates/ - - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/template-notifications/create-notification-templates/ --- # Templates @@ -103,7 +98,7 @@ Annotations can contain plain text, but you should template annotations if you n Here’s an example of templating an annotation, which explains where and why the alert was triggered. In this case, the alert triggers when CPU usage exceeds a threshold, and the `summary` annotation provides the relevant details. ``` -CPU usage for {{ index $labels "instance" }} has exceeded 80% ({{ index $values "A" }}) for the last 5 minutes. +CPU usage for {{ $labels.instance }} has exceeded 80% ({{ $values.A.Value }}) for the last 5 minutes. ``` The outcome of this template would be: @@ -120,11 +115,14 @@ For more details on how to template annotations, refer to [Template annotations [Labels](ref:labels) are used to differentiate one alert instance from all other alert instances, as the set of labels uniquely identifies an alert instance. Notification policies and silences use labels to handle alert instances. -Template labels when you need to improve or change how alerts are uniquely identified. This is helpful if the labels you get from your query aren't detailed enough. +You can also template labels based on query results. This is helpful if the labels you get from your query aren't detailed enough. For instance: + +- Add a new label to change how alerts are identified and grouped into different alert groups. +- Add a new label used by notification policies or silences to manage how the alert is handled. Here’s an example of templating a `severity` label based on the query value: -``` +```go {{ if (gt $values.A.Value 90.0) -}} critical {{ else if (gt $values.A.Value 80.0) -}} @@ -136,26 +134,23 @@ low {{- end }} ``` -Avoid using query values in labels, as this may result in the creation of numerous alerts when only one is needed. Use annotation to inform about the query value instead. - For more details on how to template labels, refer to [Template annotations and labels](ref:templating-labels-annotations). ## Template notifications -[Notification templates](ref:notification-messages) allow you to customize the content of your notifications, such as the subject of an email or the body of a Slack message. +[Notification templates](ref:template-notifications) allow you to customize the content of your notifications, such as the subject of an email or the body of a Slack message. Notification templates differ from templating annotations and labels in the following ways: - Notification templates are assigned to the **Contact point**, rather than the alert rule. - If not specified, the contact point uses a default template that includes relevant alert information. -- You can create reusable notification templates and reference them in other templates. - The same template can be shared across multiple contact points, making it easier to maintain and ensuring consistency. -- While both annotation/label templates and notification templates use the same templating language, the available variables and functions differ. For more details, refer to the [notification template reference](ref:notification-message-reference) and [annotation/label template reference](ref:templating-labels-annotations). - Notification templates should not be used to add additional information to individual alerts—use annotations for that purpose. +- While both annotation/label templates and notification templates use the same templating language, the available variables and functions differ. For more details, refer to the [notification template reference](ref:notification-message-reference) and [annotation/label template reference](ref:templating-labels-annotations). Here is an example of a notification template that summarizes all firing and resolved alerts in a notification group: -``` +```go {{ define "alerts.message" -}} {{ if .Alerts.Firing -}} {{ len .Alerts.Firing }} firing alert(s) @@ -184,4 +179,4 @@ The notification message to the contact point would look like this: - The web server web1 has been responding to 5% of HTTP requests with 5xx errors for the last 5 minutes. ``` -For instructions on creating and using notification templates, refer to [Create notification templates.](ref:create-notification-templates) +For more details, refer to [Template notifications](ref:template-notifications). diff --git a/docs/sources/shared/alerts/template-language.md b/docs/sources/shared/alerts/template-language.md new file mode 100644 index 00000000000..44af4f5668b --- /dev/null +++ b/docs/sources/shared/alerts/template-language.md @@ -0,0 +1,237 @@ +--- +title: 'Template language' +--- + +## If + +You can use `if` statements in templates. For example, you can print `Variable empty` when a variable is empty: + +```go +{{ if $element }} +Element value: {{$element}} +{{ else }} +Element is empty +{{ end }} +``` + +## With + +`with` is similar to `if` statements, but unlike `if`, it updates dot(`.`) to refer to the value of the expression in `with`: + +```go +{{ with $array }} +There are {{ len . }} item(s) +{{ else }} +There are no alerts +{{ end }} +``` + +## Range + +`range` iterates over an array or map, and dot (`.`) is set to the current element of the array: + +```go +{{ range $array }} +{{ .itemPropertyName }} +{{ end }} +``` + +Optionally, you can handle empty objects using `else`: + +```go +{{ range $array }} + {{ .itemPropertyName }} +{{ else }} + Empty array +{{ end }} +``` + +You can also get the index of each item within a range by defining index and value variables at the start of the range: + +```go +{{ $num_items := len $array }} +{{ range $index, $item := $array }} +This is item {{ $index }} out of {{ $num_items }} +{{ end }} +``` + +Additionally, you can use `{{break}}` to stop the remaining iterations or `{{continue}}` to stop the current iteration and continue with the next one. + +## Functions + +The global functions available in `text/template` are: + +| Function | Description | +| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `and` | Returns the boolean AND of its arguments by returning the first empty argument or the last argument. | +| `call` | Returns the result of calling the first argument, which must be a function, with the remaining arguments as parameters. | +| `html` | Returns the escaped HTML equivalent of the textual representation of its arguments. | +| `index` | Returns the result of indexing its first argument by the following arguments, e.g., `{{ index $labels "instance" }}` returns the `instance` key in the `$labels` map variable. | +| `slice` | Returns the result of slicing its first argument by the remaining arguments. | +| `js` | Returns the escaped JavaScript equivalent of the textual representation of its arguments. | +| `len` | Returns the integer length of its argument, e.g., `{{ len $array }}` | +| `not` | Returns the boolean negation of its single argument. | +| `or` | Returns the boolean OR of its arguments by returning the first non-empty argument or the last argument. | +| `print` | An alias for fmt.Sprint | +| `printf` | An alias for fmt.Sprintf | +| `println` | An alias for fmt.Sprintln | +| `urlquery` | Returns the escaped value of the textual representation of its arguments in a form suitable for embedding in a URL query | + +For more details, refer to the official documentation on [functions in `text/template`](https://pkg.go.dev/text/template#hdr-Functions). + +## Comparison operators + +Boolean comparison operators are also available in `text/template`: + +| Function | Description | +| -------- | ----------------------------------------- | +| `eq` | Returns the boolean truth of arg1 == arg2 | +| `ne` | Returns the boolean truth of arg1 != arg2 | +| `lt` | Returns the boolean truth of arg1 < arg2 | +| `le` | Returns the boolean truth of arg1 <= arg2 | +| `gt` | Returns the boolean truth of arg1 > arg2 | +| `ge` | Returns the boolean truth of arg1 >= arg2 | + +## Variables + +Variables in `text/template` must be created within the template. For example, you can create a variable with the current value of dot (`.`) and assign a string or another object to the variable like this: + +```go +{{ $variable := . }} +{{ $variable := "This is a test" }} +{{ $variable }} +``` + +This template outputs: + +``` +This is a test +``` + +## Templates + +You can create reusable templates that can be executed from other templates or within the same template. + +Define templates using `define` and the name of the template in double quotes: + +```go +{{ define "print_labels" }} +{{ end }} +``` + +You should not define templates with the same name as other templates, including default templates such as `__subject`, `__text_values_list`, `__text_alert_list`, `default.title` and `default.message`. Where a template has been created with the same name as a default template, or a template in another notification template, Grafana might use either template. Grafana does not prevent, or show an error message, when there are two or more templates with the same name. + +### Execute templates + +You can execute defined templates using `template`, the name of the template in double quotes, and the cursor that should be passed to the template: + +```go +{{ template "print_labels" . }} +``` + +Within a template dot refers to the value that is passed to the template. + +For example, if a template is passed a list of firing alerts then dot refers to that list of firing alerts: + +```go +{{ template "print_alerts" .Alerts }} +``` + +If the template is passed the sorted labels for an alert then dot refers to the list of sorted labels: + +```go +{{ template "print_labels" .SortedLabels }} +``` + +This is useful when writing reusable templates. For example, to print all alerts you might write the following: + +```go +{{ template "print_alerts" .Alerts }} +``` + +Then to print just the firing alerts you could write this: + +```go +{{ template "print_alerts" .Alerts.Firing }} +``` + +This works because both `.Alerts` and `.Alerts.Firing` is a list of alerts. + +```go +{{ define "print_alerts" }} +{{ range . }} +{{ template "print_labels" .SortedLabels }} +{{ end }} +{{ end }} +``` + +{{% admonition type="note" %}} +You cannot create independent, reusable templates for labels and annotations as you can with notification templates. In alert rule templates, you need to write each template inline within the label or annotation field. +{{% /admonition %}} + +## Comments + +You can add comments with `{{/*` and `*/}}`: + +```go +{{/* This is a comment */}} +``` + +To avoid adding line breaks, use: + +```go +{{- /* This is a comment with no leading or trailing line breaks */ -}} +``` + +## Indentation + +You can use indentation, both tabs and spaces, and line breaks, to make templates more readable: + +```go +{{ range .Alerts }} + {{ range .Labels.SortedPairs }} + {{ .Name }} = {{ .Value }} + {{ end }} +{{ end }} +``` + +However, indentation in the template is also present in the text. + +### Remove spaces and line breaks + +In text/template use `{{-` and `-}}` to remove leading and trailing spaces and line breaks. + +For example, when using indentation and line breaks to make a template more readable: + +```go +{{ range .Alerts }} + {{ range .Labels.SortedPairs }} + {{ .Name }} = {{ .Value }} + {{ end }} +{{ end }} +``` + +The indentation and line breaks are also present in the text: + +``` + alertname = "Test" + + grafana_folder = "Test alerts" +``` + +You can remove the indentation and line breaks from the text changing `}}` to `-}}` at the start of each range: + +```go +{{ range .Alerts -}} + {{ range .Labels.SortedPairs -}} + {{ .Name }} = {{ .Value }} + {{ end }} +{{ end }} +``` + +The indentation and line breaks in the template are now absent from the text: + +``` +alertname = "Test" +grafana_folder = "Test alerts" +```