public/grafana
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

PackageJson: Prettify markdown/mdx on commit with lint-staged (#37616)

Browse Source 
* Format md,mdx files with prettier on lint-staged

* Manually run prettier on docs/sources
This commit is contained in:
Connor Lindsey
2021-08-06 07:52:36 -06:00
committed by GitHub
parent e9c032f10f
commit b78a67cec7
301 changed files with 3216 additions and 2980 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sources/alerting/unified-alerting/_index.md
+12 -12
View File
@@ -5,9 +5,10 @@ weight = 113
+++
# Overview of Grafana 8 alerts
Alerts allow you to know about problems in your systems moments after they occur. Robust and actionable alerts help you identify and resolve issues quickly, minimizing disruption to your services.
>**Note:** This information is for the new, Grafana 8 Alerts. This is an [opt-in]({{< relref"./opt-in.md" >}}) feature released in Grafana 8.0. Grafana still supports [legacy dashboard alerts]({{< relref "../old-alerting/_index.md" >}}) out of the box
> **Note:** This information is for the new, Grafana 8 Alerts. This is an [opt-in]({{< relref"./opt-in.md" >}}) feature released in Grafana 8.0. Grafana still supports [legacy dashboard alerts]({{< relref "../old-alerting/_index.md" >}}) out of the box
Alerts have four main components:
@@ -20,7 +21,6 @@ Alerts have four main components:
You can perform the following tasks for alerts:
- [Create a Grafana managed alert rule]({{< relref "alerting-rules/create-grafana-managed-rule.md" >}})
- [Create a Cortex or Loki managed alert rule]({{< relref "alerting-rules/create-cortex-loki-managed-rule.md" >}})
- [View existing alert rules and their current state]({{< relref "alerting-rules/rule-list.md" >}})
@@ -38,6 +38,7 @@ The current alerting system doesn't support high availability. Alert notificatio
Grafana managed alerts are evaluated by the Grafana backend. Rule evaluations are scheduled, according to the alert rule configuration, and queries are evaluated by an engine that is part of core Grafana.
Alerting rules can only query backend data sources with alerting enabled:
- builtin or developed and maintained by grafana: `Graphite`, `Prometheus`, `Loki`, `InfluxDB`, `Elasticsearch`,
`Google Cloud Monitoring`, `Cloudwatch`, `Azure Monitor`, `MySQL`, `PostgreSQL`, `MSSQL`, `OpenTSDB`, `Oracle`, and `Azure Data Explorer`
- any community backend data sources with alerting enabled (`backend` and `alerting` properties are set in the [plugin.json]({{< relref "../../developers/plugins/metadata.md" >}}))
@@ -46,15 +47,14 @@ Alerting rules can only query backend data sources with alerting enabled:
The alerting engine publishes some internal metrics about itself. You can read more about how Grafana publishes [internal metrics]({{< relref "../../administration/view-server/internal-metrics.md" >}}).
Metric Name | Type | Description
---------- | ----------- | ----------
`alerting.alerts` | gauge | How many alerts by state
`alerting.request_duration_seconds` | histogram | Histogram of requests to the Alerting API
`alerting.active_configurations` | gauge | The number of active, non default Alertmanager configurations for grafana managed alerts
`alerting.rule_evaluations_total` | counter | The total number of rule evaluations
`alerting.rule_evaluation_failures_total` | counter | The total number of rule evaluation failures
`alerting.rule_evaluation_duration_seconds` | summary | The duration for a rule to execute
`alerting.rule_group_rules` | gauge | The number of rules
| Metric Name | Type | Description |
| ------------------------------------------- | --------- | ---------------------------------------------------------------------------------------- |
| `alerting.alerts` | gauge | How many alerts by state |
| `alerting.request_duration_seconds` | histogram | Histogram of requests to the Alerting API |
| `alerting.active_configurations` | gauge | The number of active, non default Alertmanager configurations for grafana managed alerts |
| `alerting.rule_evaluations_total` | counter | The total number of rule evaluations |
| `alerting.rule_evaluation_failures_total` | counter | The total number of rule evaluation failures |
| `alerting.rule_evaluation_duration_seconds` | summary | The duration for a rule to execute |
| `alerting.rule_group_rules` | gauge | The number of rules |
- [View alert rules and their current state]({{< relref "alerting-rules/rule-list.md" >}})
docs/sources/alerting/unified-alerting/alerting-rules/_index.md
+2 -1
View File
@@ -5,7 +5,8 @@ weight = 130
+++
# Create and manage alerting Rules
One or more queries and/or expressions, a condition, the frequency of evaluation, and the (optional) duration that a condition must be met before creating an alert. Alerting rules are how you express the criteria for creating an alert. Queries and expressions select and can operate on the data you wish to alert on. A condition sets the threshold that an alert must meet or exceed to create an alert. The interval specifies how frequently the rule should be evaluated. The duration, when configured, sets a period that a condition must be met or exceeded before an alert is created. Alerting rules also can contain settings for what to do when your query does not return any data, or there is an error attempting to execute the query.
One or more queries and/or expressions, a condition, the frequency of evaluation, and the (optional) duration that a condition must be met before creating an alert. Alerting rules are how you express the criteria for creating an alert. Queries and expressions select and can operate on the data you wish to alert on. A condition sets the threshold that an alert must meet or exceed to create an alert. The interval specifies how frequently the rule should be evaluated. The duration, when configured, sets a period that a condition must be met or exceeded before an alert is created. Alerting rules also can contain settings for what to do when your query does not return any data, or there is an error attempting to execute the query.
- [Create Cortex or Loki managed alert rule]({{< relref "./create-cortex-loki-managed-rule.md" >}})
- [Create Grafana managed alert rule]({{< relref "./create-grafana-managed-rule.md" >}})
docs/sources/alerting/unified-alerting/alerting-rules/create-cortex-loki-managed-rule.md
+10 -11
View File
@@ -7,10 +7,9 @@ weight = 400
# Create a Cortex or Loki managed alerting rule
Grafana allows you manage alerting rules for an external Cortex or Loki instance.
Grafana allows you manage alerting rules for an external Cortex or Loki instance.
In order for both Cortex and Loki data sources to work with Grafana 8.0 alerting, enable the ruler API by configuring their respective services. The`local` rule storage type, default for Loki, only supports viewing of rules. If you want to edit rules, then configure one of the other rule storage types. When configuring a Grafana Prometheus data source to point to Cortex, use the legacy `/api/prom` prefix, not `/prometheus`. Only single-binary mode is currently supported, and it is not possible to provide a separate URL for the ruler API.
In order for both Cortex and Loki data sources to work with Grafana 8.0 alerting, enable the ruler API by configuring their respective services. The`local` rule storage type, default for Loki, only supports viewing of rules. If you want to edit rules, then configure one of the other rule storage types. When configuring a Grafana Prometheus data source to point to Cortex, use the legacy `/api/prom` prefix, not `/prometheus`. Only single-binary mode is currently supported, and it is not possible to provide a separate URL for the ruler API.
## Add or edit a Cortex or Loki managed alerting rule
@@ -26,11 +25,11 @@ This section describes the fields you fill out to create an alert.
### Alert type
- **Alert name -** Enter a descriptive name. The name will be displayed in the alert rule list, as well as added as `alertname` label to every alert instance that is created from this rule.
- **Alert type -** Select **Cortex / Loki managed alert**.
- **Data source -** Select a Prometheus or Loki data source. Only Prometheus data sources that support Cortex ruler API will be available.
- **Namespace -** Select an existing rule namespace or click **Add new** to create a new one.
- **Group -** Select an existing group within the selected namespace or click **Add new** to create a new one. Newly created rules will be added to the end of the rule group.
- **Alert name -** Enter a descriptive name. The name will be displayed in the alert rule list, as well as added as `alertname` label to every alert instance that is created from this rule.
- **Alert type -** Select **Cortex / Loki managed alert**.
- **Data source -** Select a Prometheus or Loki data source. Only Prometheus data sources that support Cortex ruler API will be available.
- **Namespace -** Select an existing rule namespace or click **Add new** to create a new one.
- **Group -** Select an existing group within the selected namespace or click **Add new** to create a new one. Newly created rules will be added to the end of the rule group.
![Alert type section screenshot](/static/img/docs/alerting/unified/rule-edit-cortex-alert-type-8-0.png 'Alert type section screenshot')
@@ -42,7 +41,7 @@ Enter a PromQL or LogQL expression. Rule will fire if evaluation result has at l
### Conditions
- **For -** For how long the selected condition should violated before an alert enters `Firing` state. When condition threshold is violated for the first time, an alert becomes `Pending`. If the **for** time elapses and the condition is still violated, it becomes `Firing`. Else it reverts back to `Normal`.
- **For -** For how long the selected condition should violated before an alert enters `Firing` state. When condition threshold is violated for the first time, an alert becomes `Pending`. If the **for** time elapses and the condition is still violated, it becomes `Firing`. Else it reverts back to `Normal`.
![Conditions section](/static/img/docs/alerting/unified/rule-edit-cortex-conditions-8-0.png 'Conditions section screenshot')
@@ -52,11 +51,11 @@ Annotations and labels can be optionally added in the details section.
#### Annotations
Annotations are key and value pairs that provide additional meta information about the alert, for example description, summary, runbook URL. They are displayed in rule and alert details in the UI and can be used in contact type message templates. Annotations can also be templated, for example `Instance {{ $labels.instance }} down` will have the evaluated `instance` label value added for every alert this rule produces.
Annotations are key and value pairs that provide additional meta information about the alert, for example description, summary, runbook URL. They are displayed in rule and alert details in the UI and can be used in contact type message templates. Annotations can also be templated, for example `Instance {{ $labels.instance }} down` will have the evaluated `instance` label value added for every alert this rule produces.
#### Labels
Labels are key value pairs that categorize or identify an alert. Labels are used to match alerts in silences or match and groups alerts in notification policies. Labels are also shown in rule or alert details in the UI and can be used in contact type message templates. For example, it is common to add a `severity` label and then configure a separate notification policy for each severity. Or one could add a `team` label and configure team specific notification policies, or silence all alerts for a particular team.
Labels are key value pairs that categorize or identify an alert. Labels are used to match alerts in silences or match and groups alerts in notification policies. Labels are also shown in rule or alert details in the UI and can be used in contact type message templates. For example, it is common to add a `severity` label and then configure a separate notification policy for each severity. Or one could add a `team` label and configure team specific notification policies, or silence all alerts for a particular team.
![Details section](/static/img/docs/alerting/unified/rule-edit-details-8-0.png 'Details section screenshot')
docs/sources/alerting/unified-alerting/alerting-rules/create-grafana-managed-rule.md
+25 -26
View File
@@ -7,7 +7,7 @@ weight = 400
# Create a Grafana managed alerting rule
Grafana allows you to create alerting rules that query one or more data sources, reduce or transform the results and compare them to each other or to fix thresholds. These rules will be executed and notifications sent by Grafana itself.
Grafana allows you to create alerting rules that query one or more data sources, reduce or transform the results and compare them to each other or to fix thresholds. These rules will be executed and notifications sent by Grafana itself.
## Add or edit a Grafana managed alerting rule
@@ -23,15 +23,15 @@ This section describes the fields you fill out to create an alert.
### Alert type
- **Alert name -** Enter a descriptive name. The name will be displayed in the alert rule list, as well as added as `alertname` label to every alert instance that is created from this rule.
- **Alert type -** Select **Grafana managed alert**.
- **Folder -** Select a folder this alert rule will belong to. To create a new folder, click on the drop down and type in a new folder name.
- **Alert name -** Enter a descriptive name. The name will be displayed in the alert rule list, as well as added as `alertname` label to every alert instance that is created from this rule.
- **Alert type -** Select **Grafana managed alert**.
- **Folder -** Select a folder this alert rule will belong to. To create a new folder, click on the drop down and type in a new folder name.
![Alert type section screenshot](/static/img/docs/alerting/unified/rule-edit-grafana-alert-type-8-0.png 'Alert type section screenshot')
### Query
Add one or more [queries]({{< relref "../../../panels/queries.md" >}}) or [expressions]({{< relref "../../../panels/expressions.md" >}}). You can use classic condition expression to create a rule that will trigger a single alert if it's threshold is met, or use reduce and math expressions to create a multi dimensional alert rule that can trigger multiple alerts, one per matching series in the query result.
Add one or more [queries]({{< relref "../../../panels/queries.md" >}}) or [expressions]({{< relref "../../../panels/expressions.md" >}}). You can use classic condition expression to create a rule that will trigger a single alert if it's threshold is met, or use reduce and math expressions to create a multi dimensional alert rule that can trigger multiple alerts, one per matching series in the query result.
#### Rule with classic condition
@@ -41,7 +41,7 @@ You can use classic condition expression to create a rule that will trigger a si
1. Add an expression. Click on **Operation** dropdown and select **Classic condition**.
1. Add one or more conditions. For each condition you can specify operator (`AND` / `OR`), aggregation function, query letter and threshold value.
If a query returns multiple series, then the aggregation function and threshold check will be evaluated for each series.It will not track alert state **per series**. This has implications that are detailed in the scenario below.
If a query returns multiple series, then the aggregation function and threshold check will be evaluated for each series.It will not track alert state **per series**. This has implications that are detailed in the scenario below.
- Alert condition with query that returns 2 series: **server1** and **server2**
- **server1** series causes the alert rule to fire and switch to state `Firing`
@@ -67,25 +67,24 @@ See or [expressions documentation]({{< relref "../../../panels/expressions.md" >
### Conditions
- **Condition -** Select the letter of the query or expression whose result will trigger the alert rule. You will likely want to select either a `classic condition` or a `math` expression.
- **Evaluate every -** How often the rule should be evaluated, executing the defined queries and expressions. Must be no less than 10 seconds and a multiple of 10 seconds. Examples: `1m`, `30s`
- **Evaluate for -** For how long the selected condition should violated before an alert enters `Alerting` state. When condition threshold is violated for the first time, an alert becomes `Pending`. If the **for** time elapses and the condition is still violated, it becomes `Alerting`. Else it reverts back to `Normal`.
- **Condition -** Select the letter of the query or expression whose result will trigger the alert rule. You will likely want to select either a `classic condition` or a `math` expression.
- **Evaluate every -** How often the rule should be evaluated, executing the defined queries and expressions. Must be no less than 10 seconds and a multiple of 10 seconds. Examples: `1m`, `30s`
- **Evaluate for -** For how long the selected condition should violated before an alert enters `Alerting` state. When condition threshold is violated for the first time, an alert becomes `Pending`. If the **for** time elapses and the condition is still violated, it becomes `Alerting`. Else it reverts back to `Normal`.
#### No Data & Error handling
Toggle **Configure no data and error handling** switch to configure how the rule should handle cases where evaluation results in error or returns no data.
| No Data Option | Description |
| --------------- | ------------------------------------------------------------------------------------------ |
| No Data | Set alert state to `NoData` and rule state to `Normal` |
| Alerting | Set alert rule state to `Alerting` |
| Ok | Set alert rule state to `Normal` |
Toggle **Configure no data and error handling** switch to configure how the rule should handle cases where evaluation results in error or returns no data.
| No Data Option | Description |
| -------------- | ------------------------------------------------------ |
| No Data | Set alert state to `NoData` and rule state to `Normal` |
| Alerting | Set alert rule state to `Alerting` |
| Ok | Set alert rule state to `Normal` |
| Error or timeout option | Description |
| ----------------------- | --------------------------------------------------- |
| Alerting | Set alert rule state to `Alerting` |
| OK | Set alert rule state to `Normal` |
| Error or timeout option | Description |
| ----------------------- | ---------------------------------- |
| Alerting | Set alert rule state to `Alerting` |
| OK | Set alert rule state to `Normal` |
![Conditions section](/static/img/docs/alerting/unified/rule-edit-grafana-conditions-8-0.png 'Conditions section screenshot')
@@ -95,11 +94,11 @@ Annotations and labels can be optionally added in the details section.
#### Annotations
Annotations are key and value pairs that provide additional meta information about the alert, for example description, summary, runbook URL. They are displayed in rule and alert details in the UI and can be used in contact type message templates. Annotations can also be templated, for example `Instance {{ $labels.instance }} down` will have the evaluated `instance` label value added for every alert this rule produces.
Annotations are key and value pairs that provide additional meta information about the alert, for example description, summary, runbook URL. They are displayed in rule and alert details in the UI and can be used in contact type message templates. Annotations can also be templated, for example `Instance {{ $labels.instance }} down` will have the evaluated `instance` label value added for every alert this rule produces.
#### Labels
Labels are key value pairs that categorize or identify an alert. Labels are used to match alerts in silences or match and groups alerts in notification policies. Labels are also shown in rule or alert details in the UI and can be used in contact type message templates. For example, it is common to add a `severity` label and then configure a separate notification policy for each severity. Or one could add a `team` label and configure team specific notification policies, or silence all alerts for a particular team. Labels can also be templated like annotations, for example `{{ $labels.namespace }}/{{ $labels.job }}` will produce a new rule label that will have the evaluated `namespace` and `job` label value added for every alert this rule produces. The rule labels take precedence over the labels produced by the query/condition.
Labels are key value pairs that categorize or identify an alert. Labels are used to match alerts in silences or match and groups alerts in notification policies. Labels are also shown in rule or alert details in the UI and can be used in contact type message templates. For example, it is common to add a `severity` label and then configure a separate notification policy for each severity. Or one could add a `team` label and configure team specific notification policies, or silence all alerts for a particular team. Labels can also be templated like annotations, for example `{{ $labels.namespace }}/{{ $labels.job }}` will produce a new rule label that will have the evaluated `namespace` and `job` label value added for every alert this rule produces. The rule labels take precedence over the labels produced by the query/condition.
![Details section](/static/img/docs/alerting/unified/rule-edit-details-8-0.png 'Details section screenshot')
@@ -107,11 +106,11 @@ Labels are key value pairs that categorize or identify an alert. Labels are use
The following template variables are available when expanding annotations and labels.
| Name | Description |
| ------- | --------------- |
| $labels | The labels from the query or condition. For example, `{{ $labels.instance }}` and `{{ $labels.job }}`. |
| Name | Description |
| ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| $labels | The labels from the query or condition. For example, `{{ $labels.instance }}` and `{{ $labels.job }}`. |
| $values | The values of all reduce and math expressions that were evaluated for this alert rule. For example, `{{ $values.A }}`, `{{ $values.A.Labels }}` and `{{ $values.A.Value }}` where `A` is the `refID` of the expression. |
| $value | The value string of the alert instance. For example, `[ var='A' labels={instance=foo} value=10 ]`. |
| $value | The value string of the alert instance. For example, `[ var='A' labels={instance=foo} value=10 ]`. |
## Preview alerts
docs/sources/alerting/unified-alerting/alerting-rules/state-and-health.md
+6
View File
@@ -9,13 +9,17 @@ keywords = ["grafana", "alerting", "guide", "state"]
The concepts of state and health for alerting rules help you understand, at a glance, several key status indicators about your alerts. Alert state, alerting rule state, and alerting rule health are related, but they each convey subtly different information.
## Alerting rule state
Indicates whether any of the timeseries resulting from evaluation of the alerting rule are in an alerting state. Alerting rule state only requires a single alerting instance to be in a pending or firing state for the alerting rule state to not be normal.
- Normal: none of the timeseries returned are in an alerting state.
- Pending: at least one of the timeseries returned are in a pending state.
- Firing: at least one of the timeseries returned are in an alerting state.
## Alert state
Alert state is an indication of the output of the alerting evaluation engine.
- Normal: the condition for the alerting rule has evaluated to **false** for every timeseries returned by the evaluation engine.
- Alerting: the condition for the alerting rule has evaluated to **true** for at least one timeseries returned by the evaluation engine and the duration, if set, **has** been met or exceeded.
- Pending: the condition for the alerting rule has evaluated to **true** for at least one timeseries returned by the evaluation engine and the duration, if set, **has not** been met or exceeded.
@@ -23,7 +27,9 @@ Alert state is an indication of the output of the alerting evaluation engine.
- Error: There was an error encountered when attempting to evaluate the alerting rule.
## Alerting rule health
Indicates the status of alerting rule evaluation.
- Ok: the rule is being evaluated, data is being returned, and no errors have been encountered.
- Error: an error was encountered when evaluating the alerting rule.
- NoData: at least one of the timeseries returned during evaluation is in a NoData state.
docs/sources/alerting/unified-alerting/contact-points.md
+26 -25
View File
@@ -17,17 +17,19 @@ Grafana alerting UI allows you to configure contact points for the Grafana manag
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. Click **Add contact point**.
1. Enter a **Name** for the contact point
1. Enter a **Name** for the contact point
1. Select contact point type and fill out mandatory fields. **Optional settings** can be expanded for more options.
1. If you'd like this contact point to notify via multiple channels, for example both email and slack, click **New contact point type** and fill out additional contact point type details.
1. Click **Save contact point** button at the bottom of the page.
## Editing a contact point
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. Find the contact point you want to edit in the contact points table and click the **pen icon** on the right side.
1. Make any changes and click **Save contact point** button at the bottom of the page.
## Deleting a contact point
## Deleting a contact point
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. Find the contact point you want to edit in the contact points table and click the **trash can icon** on the right side.
1. A confirmation dialog will open. Click **Yes, delete**.
@@ -36,31 +38,31 @@ Grafana alerting UI allows you to configure contact points for the Grafana manag
## List of notifiers supported by Grafana
Name | Type
-----|-----
[DingDing](#dingdingdingtalk) | `dingding`
[Discord](#discord) | `discord`
[Email](#email) | `email`
[Google Hangouts Chat](#google-hangouts-chat) | `googlechat`
[Kafka](#kafka) | `kafka`
Line | `line`
Microsoft Teams | `teams`
[Opsgenie](#opsgenie) | `opsgenie`
[Pagerduty](#pagerduty) | `pagerduty`
Prometheus Alertmanager | `prometheus-alertmanager`
[Pushover](#pushover) | `pushover`
Sensu | `sensu`
[Sensu Go](#sensu-go) | `sensugo`
[Slack](#slack) | `slack`
Telegram | `telegram`
Threema | `threema`
VictorOps | `victorops`
[Webhook](#webhook) | `webhook`
[Zenduty](#zenduty) | `webhook`
| Name | Type |
| --------------------------------------------- | ------------------------- |
| [DingDing](#dingdingdingtalk) | `dingding` |
| [Discord](#discord) | `discord` |
| [Email](#email) | `email` |
| [Google Hangouts Chat](#google-hangouts-chat) | `googlechat` |
| [Kafka](#kafka) | `kafka` |
| Line | `line` |
| Microsoft Teams | `teams` |
| [Opsgenie](#opsgenie) | `opsgenie` |
| [Pagerduty](#pagerduty) | `pagerduty` |
| Prometheus Alertmanager | `prometheus-alertmanager` |
| [Pushover](#pushover) | `pushover` |
| Sensu | `sensu` |
| [Sensu Go](#sensu-go) | `sensugo` |
| [Slack](#slack) | `slack` |
| Telegram | `telegram` |
| Threema | `threema` |
| VictorOps | `victorops` |
| [Webhook](#webhook) | `webhook` |
| [Zenduty](#zenduty) | `webhook` |
## Manage contact points for an external Alertmanager
Grafana alerting UI supports managing external Alertmanager configuration. Once you add an [Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page where you can select either `Grafana` or an external Alertmanager as your data source.
Grafana alerting UI supports managing external Alertmanager configuration. Once you add an [Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page where you can select either `Grafana` or an external Alertmanager as your data source.
{{< figure max-width="40%" src="/static/img/docs/alerting/unified/contact-points-select-am-8-0.gif" caption="Select Alertmanager" >}}
@@ -68,7 +70,6 @@ Grafana alerting UI supports managing external Alertmanager configuration. Once
To edit global configuration options for an alertmanager, like SMTP server that is used by default for all email contact types:
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. In the dropdown at the top of the page, select an Alertmanager data source.
1. Click **Edit global config** button at the bottom of the page.
docs/sources/alerting/unified-alerting/grafana-managed-numeric-rule.md
+22 -22
View File
@@ -28,40 +28,40 @@ If there are string columns then those columns become labels. The name of column
For a MySQL table called "DiskSpace":
| Time | Host | Disk | PercentFree
| ----------- | --- | -----| --------
| 2021-June-7 | web1 | /etc | 3
| 2021-June-7 | web2 | /var | 4
| 2021-June-7 | web3 | /var | 8
| ... | ... | ... | ...
| Time | Host | Disk | PercentFree |
| ----------- | ---- | ---- | ----------- |
| 2021-June-7 | web1 | /etc | 3 |
| 2021-June-7 | web2 | /var | 4 |
| 2021-June-7 | web3 | /var | 8 |
| ... | ... | ... | ... |
You can query the data filtering on time, but without returning the time series to Grafana. For example, an alert that would trigger per Host, Disk when there is less than 5% free space:
```sql
SELECT Host, Disk, CASE WHEN PercentFree < 5.0 THEN PercentFree ELSE 0 END FROM (
SELECT
Host,
Disk,
Avg(PercentFree)
SELECT
Host,
Disk,
Avg(PercentFree)
FROM DiskSpace
Group By
Host,
Group By
Host,
Disk
Where __timeFilter(Time)
```
This query returns the following Table response to Grafana:
| Host | Disk | PercentFree
| --- | -----| --------
| web1 | /etc | 3
| web2 | /var | 4
| web3 | /var | 0
| Host | Disk | PercentFree |
| ---- | ---- | ----------- |
| web1 | /etc | 3 |
| web2 | /var | 4 |
| web3 | /var | 0 |
When this query is used as the **condition** in an alert rule, then the non-zero will be alerting. As a result, three alert instances are produced:
| Labels | Status
| ----------------------| ------
| {Host=web1,disk=/etc} | Alerting
| {Host=web2,disk=/var} | Alerting
| {Host=web3,disk=/var} | Normal
| Labels | Status |
| --------------------- | -------- |
| {Host=web1,disk=/etc} | Alerting |
| {Host=web2,disk=/var} | Alerting |
| {Host=web3,disk=/var} | Normal |
docs/sources/alerting/unified-alerting/message-templating/_index.md
+10 -7
View File
@@ -25,6 +25,7 @@ Grafana alerting UI allows you to configure templates for the Grafana managed al
> **Note:** Currently the configuration of the embedded Alertmanager is shared across organisations. Therefore users are advised to use the new Grafana 8 Alerts only if they have one organisation otherwise templates for the Grafana managed alerts will be visible by all organizations
### Create a template
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. Click **Add template**.
1. Fill in **Name** and **Content** fields.
@@ -32,18 +33,19 @@ Grafana alerting UI allows you to configure templates for the Grafana managed al
**Note** The template name used to reference this template in templating is not the value of the **Name** field, but the parameter to `define` tag in the content. When creating a template you can omit `define` entirely and it will be added automatically with same value as **Name** field. It's recommended to use the same name for `define` and **Name** field to avoid confusion.
<img src="/static/img/docs/alerting/unified/templates-create-8-0.png" width="600px">
### Edit a template
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. Find the template you want to edit in the templates table and click the **pen icon** on the right side.
1. Make any changes and click **Save template** button at the bottom of the page.
### Delete a template
1. In the Grafana side bar, hover your cursor over the **Alerting** (bell) icon and then click **Contact points**.
1. Find the template you want to edit in the templates table and click the **trash can icon** on the right side.
1. A confirmation dialog will open. Click **Yes, delete**.
1. A confirmation dialog will open. Click **Yes, delete**.
**Note** You are not prevented from deleting templates that are in use somewhere in contact points or other templates. Be careful!
@@ -51,23 +53,23 @@ Grafana alerting UI allows you to configure templates for the Grafana managed al
To use a template:
Enter `{{ template "templatename" . }}` into a contact point field, where `templatename` is the `define` parameter of a template.
Enter `{{ template "templatename" . }}` into a contact point field, where `templatename` is the `define` parameter of a template.
<img src="/static/img/docs/alerting/unified/contact-points-use-template-8-0.png" width="600px">
### Template examples
Here is an example of a template to render a single alert:
```
{{ define "alert" }}
[{{.Status}}] {{ .Labels.alertname }}
Labels:
{{ range .Labels.SortedPairs }}
{{ .Name }}: {{ .Value }}
{{ end }}
{{ if gt (len .Annotations) 0 }}
Annotations:
{{ range .Annotations.SortedPairs }}
@@ -85,6 +87,7 @@ Here is an example of a template to render a single alert:
```
Template to render entire notification message:
```
{{ define "message" }}
{{ if gt (len .Alerts.Firing) 0 }}
@@ -100,6 +103,6 @@ Template to render entire notification message:
## Manage templates for an external Alertmanager
Grafana alerting UI supports managing external Alertmanager configuration. Once you add an [Alertmanager data source]({{< relref "../../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page, allowing you to select either `Grafana` or an external Alertmanager data source.
Grafana alerting UI supports managing external Alertmanager configuration. Once you add an [Alertmanager data source]({{< relref "../../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page, allowing you to select either `Grafana` or an external Alertmanager data source.
{{< figure max-width="40%" src="/static/img/docs/alerting/unified/contact-points-select-am-8-0.gif" caption="Select Alertmanager" >}}
docs/sources/alerting/unified-alerting/message-templating/template-data.md
+39 -40
View File
@@ -3,41 +3,40 @@ title = "Template data"
keywords = ["grafana", "alerting", "guide", "contact point", "templating"]
+++
# Template data
Template data is passed on to [message templates]({{< relref "./_index.md" >}}) as well as sent as payload to webhook pushes.
Name | Type | Notes
------------------|----------|-----------------------------------------------------------------
Receiver | string | Name of the contact point that the notification is being sent to.
Status | string | `firing` if at least one alert is firing, otherwise `resolved`.
Alerts | Alert | List of alert objects that are included in this notification (see below).
GroupLabels | KeyValue | Labels these alerts were grouped by.
CommonLabels | KeyValue | Labels common to all the alerts included in this notification.
CommonAnnotations | KeyValue | Annotations common to all the alerts included in this notification.
ExternalURL | string | Back link to the Grafana that sent the notification. If using external Alertmanager, back link to this Alertmanager.
| Name | Type | Notes |
| ----------------- | -------- | -------------------------------------------------------------------------------------------------------------------- |
| Receiver | string | Name of the contact point that the notification is being sent to. |
| Status | string | `firing` if at least one alert is firing, otherwise `resolved`. |
| Alerts | Alert | List of alert objects that are included in this notification (see below). |
| GroupLabels | KeyValue | Labels these alerts were grouped by. |
| CommonLabels | KeyValue | Labels common to all the alerts included in this notification. |
| CommonAnnotations | KeyValue | Annotations common to all the alerts included in this notification. |
| ExternalURL | string | Back link to the Grafana that sent the notification. If using external Alertmanager, back link to this Alertmanager. |
The `Alerts` type exposes functions for filtering alerts:
* `Alerts.Firing` returns a list of firing alerts.
* `Alerts.Resolved` returns a list of resolved alerts.
- `Alerts.Firing` returns a list of firing alerts.
- `Alerts.Resolved` returns a list of resolved alerts.
## Alert
Name | Type | Notes
-------------|-----------|---------------------------------------------------------------------
Status | string | `firing` or `resolved`.
Labels | KeyValue | A set of labels attached to the alert.
Annotations | KeyValue | A set of annotations attached to the alert.
StartsAt | time.Time | Time the alert started firing.
EndsAt | time.Time | Only set if the end time of an alert is known. Otherwise set to a configurable timeout period from the time since the last alert was received.
GeneratorURL | string | A back link to Grafana or external Alertmanager.
SilenceURL | string | Link to grafana silence for with labels for this alert pre-filled. Only for Grafana managed alerts.
DashboardURL | string | Link to grafana dashboard, if alert rule belongs to one. Only for Grafana managed alerts.
PanelURL | string | Link to grafana dashboard panel, if alert rule belongs to one. Only for Grafana managed alerts.
Fingerprint | string | Fingerprint that can be used to identify the alert.
ValueString | string | A string that contains the labels and value of each reduced expression in the alert.
| Name | Type | Notes |
| ------------ | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Status | string | `firing` or `resolved`. |
| Labels | KeyValue | A set of labels attached to the alert. |
| Annotations | KeyValue | A set of annotations attached to the alert. |
| StartsAt | time.Time | Time the alert started firing. |
| EndsAt | time.Time | Only set if the end time of an alert is known. Otherwise set to a configurable timeout period from the time since the last alert was received. |
| GeneratorURL | string | A back link to Grafana or external Alertmanager. |
| SilenceURL | string | Link to grafana silence for with labels for this alert pre-filled. Only for Grafana managed alerts. |
| DashboardURL | string | Link to grafana dashboard, if alert rule belongs to one. Only for Grafana managed alerts. |
| PanelURL | string | Link to grafana dashboard panel, if alert rule belongs to one. Only for Grafana managed alerts. |
| Fingerprint | string | Fingerprint that can be used to identify the alert. |
| ValueString | string | A string that contains the labels and value of each reduced expression in the alert. |
## KeyValue
@@ -54,23 +53,23 @@ Here is an example containing two annotations:
In addition to direct access of data (labels and annotations) stored as KeyValue, there are also methods for sorting, removing and transforming.
Name | Arguments | Returns | Notes
------------|-----------|-----------------------------------------|----------------
SortedPairs | | Sorted list of key & value string pairs |
Remove | []string | KeyValue | Returns a copy of the Key/Value map without the given keys.
Names | | []string | List of label names
Values | | []string | List of label values
| Name | Arguments | Returns | Notes |
| ----------- | --------- | --------------------------------------- | ----------------------------------------------------------- |
| SortedPairs | | Sorted list of key & value string pairs |
| Remove | []string | KeyValue | Returns a copy of the Key/Value map without the given keys. |
| Names | | []string | List of label names |
| Values | | []string | List of label values |
## Functions
Some functions to transform values are also available, along with [default functions provided by Go templating](https://golang.org/pkg/text/template/#hdr-Functions).
Name | Arguments | Returns
-------------|------------------------------|----------------------------------------------
title | string | Capitalizes first character of each word.
toUpper | string | Converts all characters to upper case.
match | pattern, string | Match a string using RegExp.
reReplaceAll | pattern, replacement, string | RegExp substitution, unanchored.
join | string, []string | Concatenates the elements of the second argument to create a single string. First argument is the separator.
safeHtml | string | Marks string as HTML, not requiring auto-escaping.
stringSlice | ...string | Returns passed strings as slice of strings.
| Name | Arguments | Returns |
| ------------ | ---------------------------- | ------------------------------------------------------------------------------------------------------------ |
| title | string | Capitalizes first character of each word. |
| toUpper | string | Converts all characters to upper case. |
| match | pattern, string | Match a string using RegExp. |
| reReplaceAll | pattern, replacement, string | RegExp substitution, unanchored. |
| join | string, []string | Concatenates the elements of the second argument to create a single string. First argument is the separator. |
| safeHtml | string | Marks string as HTML, not requiring auto-escaping. |
| stringSlice | ...string | Returns passed strings as slice of strings. |
docs/sources/alerting/unified-alerting/notification-policies.md
+9 -13
View File
@@ -20,7 +20,7 @@ To access notification policy editing page, In the Grafana side bar, hover your
### Edit root notification policy
1. Click **edit** button on the top right of the root policy box.
1. Make changes and click **save** button.
1. Make changes and click **save** button.
### Add new specific policy
@@ -32,7 +32,6 @@ To add a nested policy to an existing specific policy, expand the parent policy
To edit a specific policy, find it in the specific routing table and click **Edit** button. Make your changes and click **Save policy**.
### Root policy fields
- **Default contact point -** The [contact point]({{< relref "./contact-points.md" >}}) to send notifications to that did not match any specific policy.
@@ -44,16 +43,14 @@ Group timing options
- **Group interval -** - How long to wait before sending an notification when an alert has been added to a group for which there has already been a notification. Default is 5 minutes.
- **Repeat interval -** - How long to wait before re-sending a notification after one has already been sent and no new alerts were added to the group. Default is 4 hours.
### Specific policy fields
- **Contact point -** The [contact point]({{< relref "./contact-points.md" >}}) to send notification to if alert matched this specific policy but did not match any of it's nested policies, or there were no nested specific policies.
- **Matching labels -** Rules for matching alert labels. See ["How label matching works"](#how-label-matching-works) below for details.
- **Continue matching subsequent sibling nodes -** If not enabled and an alert matches this policy but not any of it's nested policies, matching will stop and a notification will be sent to the contact point defined on this policy. If enabled, notification will be sent but alert will continue matching subsequent siblings of this policy, thus sending more than one notification. Use this if for example you want to send notification to a catch-all contact point as well as to one of more specific contact points handled by subsequent policies.
- **Override grouping** - Toggle if you want to override grouping for this policy. If toggled, you will be able to specify grouping same as for root policy described above. If not toggled, root policy grouping will be used.
- **Continue matching subsequent sibling nodes -** If not enabled and an alert matches this policy but not any of it's nested policies, matching will stop and a notification will be sent to the contact point defined on this policy. If enabled, notification will be sent but alert will continue matching subsequent siblings of this policy, thus sending more than one notification. Use this if for example you want to send notification to a catch-all contact point as well as to one of more specific contact points handled by subsequent policies.
- **Override grouping** - Toggle if you want to override grouping for this policy. If toggled, you will be able to specify grouping same as for root policy described above. If not toggled, root policy grouping will be used.
- **Override group timings** Toggle if you want to override group timings for this policy. If toggled, you will be able to specify group timings same as for root policy described above. If not toggled, root policy group timings will be used.
### How label matching works
A policy will match an alert if alert's labels match all of the "Matching Labels" specified on the policy.
@@ -63,15 +60,14 @@ A policy will match an alert if alert's labels match all of the "Matching Labels
- The **Regex** checkbox specifies if the inputted **Value** should be matched against labels as a regular expression. The regular expression is always anchored. If not selected it is an exact string match.
- The **Equal** checkbox specifies if the match should include alert instances that match or do not match. If not checked, the silence includes alert instances _do not_ match.
## Example setup
One usage example would be:
* Create a "default" contact point for most alerts with a non invasive contact point type, like a slack message, and set it on root policy
* Edit root policy grouping to group alerts by `cluster`, `namespace` and `alertname` so you get a notification per alert rule and specific k8s cluster & namespace.
* Create specific route for alerts coming from development cluster with an appropriate contact point
* Create a specific route for alerts with "critical" severity with a more invasive contact point type, like pager duty notification
* Create specific routes for particular teams that handle their own onduty rotations
- Create a "default" contact point for most alerts with a non invasive contact point type, like a slack message, and set it on root policy
- Edit root policy grouping to group alerts by `cluster`, `namespace` and `alertname` so you get a notification per alert rule and specific k8s cluster & namespace.
- Create specific route for alerts coming from development cluster with an appropriate contact point
- Create a specific route for alerts with "critical" severity with a more invasive contact point type, like pager duty notification
- Create specific routes for particular teams that handle their own onduty rotations
![Notification policies screenshot](/static/img/docs/alerting/unified/notification-policies-8-0.png 'Notification policies screenshot')
docs/sources/alerting/unified-alerting/opt-in.md
+5 -4
View File
@@ -8,11 +8,11 @@ weight = 128
Setting the `ngalert` feature toggle enables the new Grafana 8 alerting system.
>**Note:** We recommend that you backup Grafana's database before enabling this feature. If you are using PostgreSQL as the backend data source, then the minimum required version is 9.5.
> **Note:** We recommend that you backup Grafana's database before enabling this feature. If you are using PostgreSQL as the backend data source, then the minimum required version is 9.5.
At startup, when [the feature toggle is enabled]({{< relref "../../administration/configuration.md">}}#feature_toggles), the legacy Grafana dashboard alerting is disabled and existing dashboard alerts are migrated into a format that is compatible with the Grafana 8 alerting system. You can view these migrated rules, alongside any new alerts you create after the migration, from the Alerting page of your Grafana instance.
>**Note:** Since the new system stores the notification log and silences on disk, we require the use of persistent disks for using Grafana 8 alerts. Otherwise, the silences and notification log will get lost on a restart, and you might get unwanted or duplicate notifications.
> **Note:** Since the new system stores the notification log and silences on disk, we require the use of persistent disks for using Grafana 8 alerts. Otherwise, the silences and notification log will get lost on a restart, and you might get unwanted or duplicate notifications.
Read and write access to dashboard alerts in Grafana versions 7 and earlier were governed by the dashboard and folder permissions under which the alerts were stored. In Grafana 8, alerts are stored in folders and inherit the permissions of those folders. During the migration, dashboard alert permissions are matched to the new rules permissions as follows:
@@ -20,7 +20,7 @@ Read and write access to dashboard alerts in Grafana versions 7 and earlier were
- If there are no dashboard permissions and the dashboard is under a folder, then the rule is linked to this folder and inherits its permissions.
- If there are no dashboard permissions and the dashboard is under the General folder, then the rule is linked to the `General Alerting` folder and the rule inherits the default permissions.
During beta, Grafana 8 alerting system can retrieve rules from all available Prometheus, Loki, and Alertmanager data sources. It might not be able to fetch rules from all other supported data sources at this time.
During beta, Grafana 8 alerting system can retrieve rules from all available Prometheus, Loki, and Alertmanager data sources. It might not be able to fetch rules from all other supported data sources at this time.
Also notification channels are migrated to an Alertmanager configuration with the appropriate routes and receivers. Default notification channels are added as contact points to the default route. Notification channels not associated with any Dashboard alert go to the `autogen-unlinked-channel-recv` route.
@@ -28,6 +28,7 @@ Since `Hipchat` and `Sensu` are discontinued, they are not migrated to the new a
Finally, silences (expiring after one year) are created for all paused dashboard alerts.
## Disabling Grafana 8 Alerting after migration
To disable Grafana 8 Alerting, remove or disable the `ngalert` feature toggle. Dashboard alerts will be re-enabled and any alerts created during or after the migration are deleted.
>**Note:** Any alerting rules created in the Grafana 8 Alerting system will be lost when migrating back to dashboard alerts
> **Note:** Any alerting rules created in the Grafana 8 Alerting system will be lost when migrating back to dashboard alerts
docs/sources/alerting/unified-alerting/silences.md
+3 -3
View File
@@ -23,7 +23,7 @@ To add a silence:
1. Click the **New Silence** button.
1. Select the start and end date in **Silence start and end** to indicate when the silence should go into effect and expire.
1. Optionally, update the **Duration** to alter the time for the end of silence in the previous step to correspond to the start plus the duration.
1. Enter one or more *Matching Labels* by filling out the **Name** and **Value** fields. Matchers determine which rules the silence will apply to.
1. Enter one or more _Matching Labels_ by filling out the **Name** and **Value** fields. Matchers determine which rules the silence will apply to.
1. Enter a **Comment**.
1. Enter the name of the owner in **Creator**.
1. Click **Create**.
@@ -46,12 +46,12 @@ Alert instances that have labels that match all of the "Matching Labels" specifi
## Manage silences for an external Alertmanager
Grafana alerting UI supports managing external Alertmanager silences. Once you add an [Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page where you can select either `Grafana` or an external Alertmanager as your data source.
Grafana alerting UI supports managing external Alertmanager silences. Once you add an [Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}), a dropdown displays at the top of the page where you can select either `Grafana` or an external Alertmanager as your data source.
## Create a URL to silence form with defaults filled in
When linking to silence form, you can provide default matching labels and comment via `matchers` and `comment` query parameters. `matchers` expects one more matching labels of type `[label][operator][value]` joined by a comma. `operator` can be one of `=` (equals, not regex), `!=` (not equals, not regex), `=~` (equals, regex), `!~` (not equals, not regex).
For example, to link to silence form with matching labels `severity=critical` & `cluster!~europe-.*` and comment `Silence critical EU alerts`, create a URL `https://mygrafana/aleting/silence/new?matchers=severity%3Dcritical%2Ccluster!~europe-*&comment=Silence%20critical%20EU%20alert`.
For example, to link to silence form with matching labels `severity=critical` & `cluster!~europe-.*` and comment `Silence critical EU alerts`, create a URL `https://mygrafana/aleting/silence/new?matchers=severity%3Dcritical%2Ccluster!~europe-*&comment=Silence%20critical%20EU%20alert`.
To link to a new silence page for an [external Alertmanager]({{< relref "../../datasources/alertmanager.md" >}}), add a `alertmanager` query parameter with the Alertmanager data source name.