Docs: Refactor alerting documentation (part 1) (#48664)
* Initial commit * Moved files, ad fixed broken relrefs. * Fixed other broken relrefs * More changes. * Fixing broken relrefs * More changes. * Fixed last of the broken links * More re-org. * Added aliases and some weight adjustments * More aliases. * Fix fundamentals topic. * Fixed remaining metadata issues * Ran prettier
This commit is contained in:
achatterjee-grafana
@@ -0,0 +1,13 @@
|
||||
+++
|
||||
title = "Alerting fundamentals"
|
||||
weight = 110
|
||||
aliases = ["/docs/grafana/latest/alerting/metrics/", "/docs/grafana/latest/alerting/unified-alerting/fundamentals/"]
|
||||
+++
|
||||
|
||||
# Alerting fundamentals
|
||||
|
||||
This section covers the fundamental concepts of Grafana 8 alerting.
|
||||
|
||||
- [Alertmanager]({{< relref "./alertmanager.md" >}})
|
||||
- [State and health of alerting rules]({{< relref "./state-and-health.md" >}})
|
||||
- [Evaluating Grafana managed alerts]({{< relref "./evaluate-grafana-alerts.md" >}})
|
||||
@@ -0,0 +1,39 @@
|
||||
+++
|
||||
title = "Alertmanager"
|
||||
aliases = ["/docs/grafana/latest/alerting/metrics/", "/docs/grafana/latest/alerting/unified-alerting/fundamentals/alertmanager/"]
|
||||
weight = 116
|
||||
+++
|
||||
|
||||
# Alertmanager
|
||||
|
||||
The Alertmanager helps both group and manage alert rules, adding a layer of orchestration on top of the alerting engines. To learn more, see [Prometheus Alertmanager documentation](https://prometheus.io/docs/alerting/latest/alertmanager/).
|
||||
|
||||
Grafana includes built-in support for Prometheus Alertmanager. By default, notifications for Grafana managed alerts are handled by the embedded Alertmanager that is part of core Grafana. You can configure the Alertmanager's contact points, notification policies, silences, and templates from the alerting UI by selecting the `Grafana` option from the Alertmanager drop-down.
|
||||
|
||||
> **Note:** Before v8.2, the configuration of the embedded Alertmanager was shared across organizations. If you are on an older Grafana version, we recommend that you use Grafana alerts only if you have one organization. Otherwise, your contact points are visible to all organizations.
|
||||
|
||||
Grafana alerting added support for external Alertmanager configuration. When you add an [Alertmanager data source]({{< relref "../../datasources/alertmanager.md" >}}), the Alertmanager drop-down shows a list of available external Alertmanager data sources. Select a data source to create and manage alerting for standalone Grafana Mimir or Loki data sources.
|
||||
|
||||
{{< figure max-width="40%" src="/static/img/docs/alerting/unified/contact-points-select-am-8-0.gif" max-width="250px" caption="Select Alertmanager" >}}
|
||||
|
||||
You can configure one or several external Alertmanagers to receive alerts from Grafana. Once configured, both the embedded Alertmanager **and** any configured external Alertmanagers will receive _all_ alerts.
|
||||
|
||||
You can do the setup in the "Admin" tab within the Grafana v8 Alerts UI.
|
||||
|
||||
### Add a new external Alertmanager
|
||||
|
||||
1. In the Grafana menu, click the Alerting (bell) icon to open the Alerting page listing existing alerts.
|
||||
2. Click **Admin** and then scroll down to the External Alertmanager section.
|
||||
3. Click **Add Alertmanager** and a modal opens.
|
||||
4. Add the URL and the port for the external Alertmanager. You do not need to specify the path suffix, for example, `/api/v(1|2)/alerts`. Grafana automatically adds this.
|
||||
|
||||
The external URL is listed in the table with a pending status. Once Grafana verifies that the Alertmanager is discovered, the status changes to active. No requests are made to the external Alertmanager at this point; the verification signals that alerts are ready to be sent.
|
||||
|
||||
### Edit an external Alertmanager
|
||||
|
||||
1. Click the pen symbol to the right of the Alertmanager row in the table.
|
||||
2. When the edit modal opens, you can view all the URLs that were added.
|
||||
|
||||
The edited URL will be pending until Grafana verifies it again.
|
||||
|
||||
{{< figure max-width="40%" src="/static/img/docs/alerting/unified/ext-alertmanager-active.png" max-width="650px" caption="External Alertmanagers" >}}
|
||||
@@ -0,0 +1,99 @@
|
||||
+++
|
||||
title = "Alerting on numeric data"
|
||||
aliases = ["/docs/grafana/latest/alerting/metrics/", "/docs/grafana/latest/alerting/unified-alerting/fundamentals/evaluate-grafana-alerts/"]
|
||||
weight = 116
|
||||
+++
|
||||
|
||||
# Alerting on numeric data
|
||||
|
||||
This topic describes how Grafana managed alerts are evaluated by the backend engine as well as how Grafana handles alerting on numeric rather than time series data.
|
||||
|
||||
- [Alerting on numeric data](#alerting-on-numeric-data)
|
||||
- [Alert evaluation](#alert-evaluation)
|
||||
- [Metrics from the alerting engine](#metrics-from-the-alerting-engine)
|
||||
- [Alerting on numeric data](#alerting-on-numeric-data-1)
|
||||
- [Tabular Data](#tabular-data)
|
||||
- [Example](#example)
|
||||
|
||||
## Alert evaluation
|
||||
|
||||
Grafana managed alerts query the following backend data sources that have alerting enabled:
|
||||
|
||||
- built-in data sources or those developed and maintained by Grafana: `Graphite`, `Prometheus`, `Loki`, `InfluxDB`, `Elasticsearch`,
|
||||
`Google Cloud Monitoring`, `Cloudwatch`, `Azure Monitor`, `MySQL`, `PostgreSQL`, `MSSQL`, `OpenTSDB`, `Oracle`, and `Azure Monitor`
|
||||
- community developed backend data sources with alerting enabled (`backend` and `alerting` properties are set in the [plugin.json]({{< relref "../../developers/plugins/metadata.md" >}}))
|
||||
|
||||
### Metrics from the alerting engine
|
||||
|
||||
The alerting engine publishes some internal metrics about itself. You can read more about how Grafana publishes [internal metrics]({{< relref "../../administration/view-server/internal-metrics.md" >}}). See also, [View alert rules and their current state]({{< relref "../alerting-rules/rule-list.md" >}}).
|
||||
|
||||
| Metric Name | Type | Description |
|
||||
| ------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------- |
|
||||
| `grafana_alerting_alerts` | gauge | How many alerts by state |
|
||||
| `grafana_alerting_request_duration` | histogram | Histogram of requests to the Alerting API |
|
||||
| `grafana_alerting_active_configurations` | gauge | The number of active, non default Alertmanager configurations for grafana managed alerts |
|
||||
| `grafana_alerting_rule_evaluations_total` | counter | The total number of rule evaluations |
|
||||
| `grafana_alerting_rule_evaluation_failures_total` | counter | The total number of rule evaluation failures |
|
||||
| `grafana_alerting_rule_evaluation_duration` | summary | The duration for a rule to execute |
|
||||
| `grafana_alerting_rule_group_rules` | gauge | The number of rules |
|
||||
|
||||
## Alerting on numeric data
|
||||
|
||||
Among certain data sources numeric data that is not time series can be directly alerted on, or passed into Server Side Expressions (SSE). This allows for more processing and resulting efficiency within the data source, and it can also simplify alert rules.
|
||||
When alerting on numeric data instead of time series data, there is no need to reduce each labeled time series into a single number. Instead labeled numbers are returned to Grafana instead.
|
||||
|
||||
### Tabular Data
|
||||
|
||||
This feature is supported with backend data sources that query tabular data:
|
||||
|
||||
- SQL data sources such as MySQL, Postgres, MSSQL, and Oracle.
|
||||
- The Azure Kusto based services: Azure Monitor (Logs), Azure Monitor (Azure Resource Graph), and Azure Data Explorer.
|
||||
|
||||
A query with Grafana managed alerts or SSE is considered numeric with these data sources, if:
|
||||
|
||||
- The "Format AS" option is set to "Table" in the data source query.
|
||||
- The table response returned to Grafana from the query includes only one numeric (e.g. int, double, float) column, and optionally additional string columns.
|
||||
|
||||
If there are string columns then those columns become labels. The name of column becomes the label name, and the value for each row becomes the value of the corresponding label. If multiple rows are returned, then each row should be uniquely identified their labels.
|
||||
|
||||
### Example
|
||||
|
||||
For a MySQL table called "DiskSpace":
|
||||
|
||||
| Time | Host | Disk | PercentFree |
|
||||
| ----------- | ---- | ---- | ----------- |
|
||||
| 2021-June-7 | web1 | /etc | 3 |
|
||||
| 2021-June-7 | web2 | /var | 4 |
|
||||
| 2021-June-7 | web3 | /var | 8 |
|
||||
| ... | ... | ... | ... |
|
||||
|
||||
You can query the data filtering on time, but without returning the time series to Grafana. For example, an alert that would trigger per Host, Disk when there is less than 5% free space:
|
||||
|
||||
```sql
|
||||
SELECT Host, Disk, CASE WHEN PercentFree < 5.0 THEN PercentFree ELSE 0 END FROM (
|
||||
SELECT
|
||||
Host,
|
||||
Disk,
|
||||
Avg(PercentFree)
|
||||
FROM DiskSpace
|
||||
Group By
|
||||
Host,
|
||||
Disk
|
||||
Where __timeFilter(Time)
|
||||
```
|
||||
|
||||
This query returns the following Table response to Grafana:
|
||||
|
||||
| Host | Disk | PercentFree |
|
||||
| ---- | ---- | ----------- |
|
||||
| web1 | /etc | 3 |
|
||||
| web2 | /var | 4 |
|
||||
| web3 | /var | 0 |
|
||||
|
||||
When this query is used as the **condition** in an alert rule, then the non-zero will be alerting. As a result, three alert instances are produced:
|
||||
|
||||
| Labels | Status |
|
||||
| --------------------- | -------- |
|
||||
| {Host=web1,disk=/etc} | Alerting |
|
||||
| {Host=web2,disk=/var} | Alerting |
|
||||
| {Host=web3,disk=/var} | Normal |
|
||||
@@ -0,0 +1,30 @@
|
||||
+++
|
||||
title = "State and health of alerting rules"
|
||||
description = "State and Health of alerting rules"
|
||||
keywords = ["grafana", "alerting", "guide", "state"]
|
||||
aliases = ["/docs/grafana/llatest/alerting/unified-alerting/alerting-rules/state-and-health/"]
|
||||
+++
|
||||
|
||||
# State and health of alerting rules
|
||||
|
||||
The state and health of alerting rules help you understand several key status indicators about your alerts. There are three key components: alert state, alerting rule state, and alerting rule health. Although related, each component conveys subtly different information.
|
||||
|
||||
## Alerting rule state
|
||||
|
||||
- **Normal**: None of the time series returned by the evaluation engine is in a Pending or Firing state.
|
||||
- **Pending**: At least one time series returned by the evaluation engine is Pending.
|
||||
- **Firing**: At least one time series returned by the evaluation engine is Firing.
|
||||
|
||||
## Alert state
|
||||
|
||||
- **Normal**: Condition for the alerting rule is **false** for every time series returned by the evaluation engine.
|
||||
- **Alerting**: Condition of the alerting rule is **true** for at least one time series returned by the evaluation engine. The duration for which the condition must be true before an alert fires, if set, is met or has exceeded.
|
||||
- **Pending**: Condition of the alerting rule is **true** for at least one time series returned by the evaluation engine. The duration for which the condition must be true before an alert fires, if set, **has not** been met.
|
||||
- **NoData**: the alerting rule has not returned a time series, all values for the time series are null, or all values for the time series are zero.
|
||||
- **Error**: Error when attempting to evaluate an alerting rule.
|
||||
|
||||
## Alerting rule health
|
||||
|
||||
- **Ok**: No error when evaluating an alerting rule.
|
||||
- **Error**: Error when evaluating an alerting rule.
|
||||
- **NoData**: The absence of data in at least one time series returned during a rule evaluation.
|
||||
Reference in New Issue
Block a user