From 8fe36e1c837fd381cde100790445aa0f624f1a98 Mon Sep 17 00:00:00 2001 From: Pepe Cano <825430+ppcano@users.noreply.github.com> Date: Tue, 3 Dec 2024 15:57:17 +0100 Subject: [PATCH] Alerting docs: clarify `$labels` and query labels (#97350) * Alerting docs: clarify `$labels` and query labels * fix mispell --- .../alerting-rules/templates/_index.md | 22 +++++++++---------- .../alerting-rules/templates/examples.md | 10 +++++---- .../alerting-rules/templates/reference.md | 11 +++++++--- .../template-notifications/reference.md | 7 +++++- .../alert-rules/annotation-label.md | 8 ++++--- 5 files changed, 36 insertions(+), 22 deletions(-) diff --git a/docs/sources/alerting/alerting-rules/templates/_index.md b/docs/sources/alerting/alerting-rules/templates/_index.md index 77ef6073989..d5ce1572772 100644 --- a/docs/sources/alerting/alerting-rules/templates/_index.md +++ b/docs/sources/alerting/alerting-rules/templates/_index.md @@ -18,16 +18,16 @@ labels: title: Template annotations and labels weight: 500 refs: - labels: + reference-labels: - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/fundamentals/alert-rules/annotation-label/#labels + destination: /docs/grafana//alerting/alerting-rules/templates/reference/#labels - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/#labels - values: + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/#labels + reference-values: - pattern: /docs/grafana/ - destination: /docs/grafana//alerting/fundamentals/alert-rules/annotation-label/#values + destination: /docs/grafana//alerting/alerting-rules/templates/reference/#values - pattern: /docs/grafana-cloud/ - destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/#values + destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/templates/reference/#values annotations: - pattern: /docs/grafana/ destination: /docs/grafana//alerting/fundamentals/alert-rules/annotation-label/#annotations @@ -92,7 +92,7 @@ Refer to [Templates Introduction](ref:intro-to-templates) for a more detailed ex Both types of templates are written in the Go templating system. However, it's important to understand that variables and functions used in notification templates are different from those used in annotation and label templates. -1. **Template annotations and labels**: These templates add extra information to individual alert instances. Template variables like [`$labels`](ref:labels) and [`$values`](ref:values) represent alert query data of the individual alert instance. +1. **Template annotations and labels**: These templates add extra information to individual alert instances. Template variables like [`$labels`](ref:reference-labels) and [`$values`](ref:reference-values) represent alert query data of the individual alert instance. 1. **Template notifications**: Notification templates format the notification content for a group of alerts. Variables like [`.Alerts`](ref:notification-data-reference) include all firing and resolved alerts in the notification. ## Template annotations @@ -111,11 +111,11 @@ CPU usage has exceeded 80% for the last 5 minutes. However, if you want to display dynamic query values in annotations, you need to use template code. Common use cases include: -- Displaying the query value or threshold that triggered the alert. -- Highlighting label information that identifies the alert, such as environment, region, or priority. +- Displaying the query value that triggered the alert. +- Highlighting label information that identifies the alert, such as the environment, instance, or region. - Providing specific instructions based on query values. -- Customizing runbook links depending on query or label values. -- Including contact information based on alert labels. +- Customizing runbook links depending on query labels. +- Including contact information based on query labels. For instance, you can template the previous example to display the specific instance and CPU value that triggered the alert. diff --git a/docs/sources/alerting/alerting-rules/templates/examples.md b/docs/sources/alerting/alerting-rules/templates/examples.md index 47e562544a2..e31cda2f626 100644 --- a/docs/sources/alerting/alerting-rules/templates/examples.md +++ b/docs/sources/alerting/alerting-rules/templates/examples.md @@ -115,7 +115,9 @@ CPU usage has exceeded 80% (81.2345) for the last 5 minutes. ### Include labels for extra details -To provide additional context, you can include labels from the query. For instance, access the [`$labels`](ref:reference-labels) variable to display a label that informs about the affected instance: +To provide additional context, you can include labels from the query using the [`$labels`](ref:reference-labels) variable. + +For instance, the previous case could also include the affected `instance`. ``` CPU usage for {{ $labels.instance }} has exceeded 80% ({{ $values.A.Value }}) for the last 5 minutes. @@ -125,14 +127,14 @@ CPU usage for {{ $labels.instance }} has exceeded 80% ({{ $values.A.Value }}) fo CPU usage for Instance 1 has exceeded 80% (81.2345) for the last 5 minutes. ``` -Annotations can also be used to provide a summary of key alert labels, such as the environment and alert severity. For instance, you can display a summary of the alert with important labels like: +You can incorporate any labels returned by the query into the template. For instance, the following template includes information about the environment and region where the alert occurred. ``` -Alert triggered in {{ $labels.environment }} with severity {{ $labels.severity }} +Alert triggered in {{ $labels.environment }} on {{ $labels.region }} region. ``` ```template_output -Alert triggered in production with severity critical. +Alert triggered in production on AMER region. ``` ### Print a range query diff --git a/docs/sources/alerting/alerting-rules/templates/reference.md b/docs/sources/alerting/alerting-rules/templates/reference.md index f33a8632cb3..8df98144878 100644 --- a/docs/sources/alerting/alerting-rules/templates/reference.md +++ b/docs/sources/alerting/alerting-rules/templates/reference.md @@ -16,6 +16,11 @@ title: Annotation and label template reference menuTitle: Template reference weight: 101 refs: + label-types: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/fundamentals/alert-rules/annotation-label/#label-types + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/#label-types notification-template-reference: - pattern: /docs/grafana/ destination: /docs/grafana//alerting/configure-notifications/template-notifications/reference/ @@ -72,15 +77,15 @@ The following variables are available when templating annotations and labels: | Variables | Description | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| [$labels](#labels) | Contains all labels from the query. | +| [$labels](#labels) | Contains all labels from the query, only query labels. | | [$values](#values) | Contains the labels and floating point values of all instant queries and expressions, indexed by their Ref IDs. | | [$value](#value) | A string containing the labels and values of all instant queries; threshold, reduce and math expressions, and classic conditions in the alert rule. | ### $labels -The `$labels` variable contains all labels from the query. +The `$labels` variable contains all labels from the query. It excludes [user-configured and reserved labels](ref:label-types), containing only query labels. -{{< figure src="/media/docs/alerting/query-labels-and_value.png" max-width="1200px" caption="An alert rule displaying labels and value from a query" >}} +{{< figure src="/media/docs/alerting/query-labels-and-values.png" max-width="1200px" caption="An alert rule displaying labels and value from a query." >}} For example, suppose you have a query that returns CPU usage for all of your servers, and you have an alert rule that fires when any of your servers have exceeded 80% CPU usage for the last 5 minutes. You want to add a summary annotation to the alert that tells you which server is experiencing high CPU usage. With the `$labels` variable you can write a template that prints a human-readable sentence such as: diff --git a/docs/sources/alerting/configure-notifications/template-notifications/reference.md b/docs/sources/alerting/configure-notifications/template-notifications/reference.md index e8921ab6bea..501b1e5331f 100644 --- a/docs/sources/alerting/configure-notifications/template-notifications/reference.md +++ b/docs/sources/alerting/configure-notifications/template-notifications/reference.md @@ -17,6 +17,11 @@ title: Notification template reference menuTitle: Template reference weight: 102 refs: + label-types: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/fundamentals/alert-rules/annotation-label/#label-types + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/#label-types alert-rule-template-reference: - pattern: /docs/grafana/ destination: /docs/grafana//alerting/alerting-rules/templates/reference/ @@ -88,7 +93,7 @@ Here's an example that prints all available notification data from dot (`.`): | Name | Type | Description | | -------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `Status` | string | Firing or resolved. | -| `Labels` | [KV](#kv) | The labels for this alert. | +| `Labels` | [KV](#kv) | The labels for this alert. It includes all [types of labels](ref:label-types). | | `Annotations` | [KV](#kv) | The annotations for this alert. | | `StartsAt` | [Time](#time) | The time the alert fired | | `EndsAt` | [Time](#time) | Only set if the end time of an alert is known. Otherwise set to a configurable timeout period from the time since the last alert was received. | diff --git a/docs/sources/alerting/fundamentals/alert-rules/annotation-label.md b/docs/sources/alerting/fundamentals/alert-rules/annotation-label.md index 9640d3c3fe0..41d3c4c0220 100644 --- a/docs/sources/alerting/fundamentals/alert-rules/annotation-label.md +++ b/docs/sources/alerting/fundamentals/alert-rules/annotation-label.md @@ -87,11 +87,13 @@ Labels that you manually configure in the alert rule to identify the generated a Additionally, you can use a [template](ref:templates) to customize the label value and generate dynamic values from query data. -**Data source query labels** +**Query labels** -For example, if you are monitoring temperature readings and each time series for these readings has a `sensor_id`, and a `location` label, an alert instance might have the labels `{sensor_id="1",location="NY"}`, while another alert instance might have `{sensor_id="2",location="WA"}`. +Query labels are labels returned by the data source query. -Data source query labels labels are also used to generate multiple alert instances from the same alert rule, helping to distinguish alerts from different data. +{{< figure src="/media/docs/alerting/query-labels-and-values.png" max-width="1200px" caption="An alert rule query returning labels from the query." >}} + +Query labels can generate multiple alert instances from the same alert rule, helping to distinguish alerts from different data. In this example, the `instance` label generates an alert instance for each server. **Reserved labels**