docs(alerting): MissingSeriesEvalsToResolve and New Stale alert instances docs page (#105415)

* docs(alerting): New `Stale alert instances` docs page and document the `Missing series evaluations to resolve` option.

* update meta description
This commit is contained in:
Pepe Cano
2025-05-15 13:36:59 +02:00
committed by GitHub
parent d75a930548
commit bb04b9c55b
7 changed files with 130 additions and 58 deletions
@@ -24,11 +24,17 @@ refs:
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/use-dashboards/#time-units-and-relative-ranges
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/visualizations/dashboards/use-dashboards/#time-units-and-relative-ranges
configure-missing-series-evaluations-to-resolve:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/#configure-missing-series-evaluations-to-resolve
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/#configure-missing-series-evaluations-to-resolve
alert-instance-state:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/#alert-instance-state
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/#alert-instance-state
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/
recovery-threshold:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#recovery-threshold
@@ -44,6 +50,11 @@ refs:
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#pending-period
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/#pending-period
keep-firing-for:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#keep-firing-for
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/#keep-firing-for
alert-rule-evaluation:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/
@@ -241,16 +252,16 @@ To do this, you need to make sure that your alert rule is in the right evaluatio
After a condition is met, the alert goes into the **Pending** state.
If the condition remains active for the duration specified, the alert transitions to the **Firing** state, else it reverts to the **Normal** state.
1. Optionally, set the [Keep firing for](ref:keep-firing-for) period.
You can set the minimum amount of time that an alert remains firing after the breached threshold expression no longer returns any results. This sets an alert to a "Recovering" state for the duration of time set here. The Recovering state can be used to reduce noise from flapping alerts. Select "none" stop an alert from firing immediately after the breach threshold is cleared.
1. Turn on pause alert notifications, if required.
You can pause alert rule evaluation to prevent noisy alerting while tuning your alerts.
Pausing stops alert rule evaluation and doesn't create any alert instances.
This is different to [mute timings](ref:mute-timings), which stop notifications from being delivered, but still allows for alert rule evaluation and the creation of alert instances.
1. Set the time threshold for alerts firing.
You can set the minimum amount of time that an alert remains firing after the breached threshold expression no longer returns any results. This sets an alert to a "Recovering" state for the duration of time set here. The Recovering state can be used to reduce noise from flapping alerts. Select "none" stop an alert from firing immediately after the breach threshold is cleared.
1. In **Configure no data and error handling**, you can define the alerting behavior and alerting state for two scenarios:
- When the evaluation returns **No data** or all values are null.
@@ -262,6 +273,8 @@ You can set the minimum amount of time that an alert remains firing after the br
For more details, refer to [alert instance states](ref:alert-instance-state) and [modify the no data or error state](ref:modify-the-no-data-or-error-state).
1. In **Configure no data and error handling**, you can also configure [Missing series evaluations to resolve](ref:configure-missing-series-evaluations-to-resolve): how many consecutive evaluation intervals must pass without data before an alert instance is considered stale.
## Configure notifications
Choose to select a contact point directly from the alert rule form or to use notification policy routing as well as set up mute timings and groupings.
@@ -20,9 +20,9 @@ weight: 500
refs:
shared-stale-alert-instances:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/#stale-alert-instances-missingseries
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/#stale-alert-instances-missingseries
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/
reference-labels:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/templates/reference/#labels
@@ -18,9 +18,9 @@ weight: 102
refs:
shared-stale-alert-instances:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/#stale-alert-instances-missingseries
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/#stale-alert-instances-missingseries
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/
labels:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#labels
@@ -0,0 +1,79 @@
---
canonical: https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/
description: An alert instance is considered stale when its series disappears for a number of consecutive evaluation intervals. Learn how Grafana resolves them.
keywords:
- grafana
- alerting
- guide
- state
labels:
products:
- cloud
- enterprise
- oss
title: Stale alert instances
weight: 110
refs:
no-data-state:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/#no-data-state
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/#no-data-state
no-data-and-error-handling:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/#configure-no-data-and-error-handling
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-grafana-managed-rule/#configure-no-data-and-error-handling
guide-missing-data:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/learn/missing-data/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/learn/missing-data/
---
# Stale alert instances
An alert instance is considered **stale** if the alert rule query returns data but its dimension (or series) has disappeared for a number of consecutive evaluation intervals (2 by default).
This is different from the [**No Data** state](ref:no-data-state), which occurs when the alert rule query runs successfully but returns no dimensions (or series) at all.
A stale alert instance transitions to the **Normal (MissingSeries)** state as **Resolved**, and is then evicted:
| Eval. Interval | 1 | 2 | 3 | 4 | 5 |
| :--------------- | :-- | :-------------- | :-------------- | :-- | :-- |
| Alert instance A | ✔ | ✔ | ✔ | ✔ | ✔ |
| Alert instance B | ✔ | `MissingSeries` | `MissingSeries` | ️📩 | |
{{< admonition type="note" >}}
Stale alert instances are supported only for Grafana-managed alert rules.
{{< /admonition >}}
## How Grafana handles stale alert instances
The process for handling stale alert instances is as follows:
1. The alert rule runs and returns data for some label sets.
1. An alert instance that previously existed is now missing.
1. Grafana keeps the previous state of the alert instance for the number of evaluation intervals specified in [Missing series evaluations to resolve](#configure-missing-series-evaluations-to-resolve).
1. If it remains missing after two intervals, it transitions to the **Normal** state and sets **MissingSeries** in the `grafana_state_reason` annotation.
1. Stale alert instances in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**, and are routed for notifications like other resolved alerts.
1. The alert instance is removed from the UI.
{{< admonition type="tip" >}}
For common examples and practical guidance on handling **No Data** and **stale** alert scenarios, see [Handling missing data](ref:guide-missing-data).
{{< /admonition >}}
## Configure Missing series evaluations to resolve
In [Configure no data and error handling > Missing series evaluations to resolve](ref:no-data-and-error-handling), you can set how many consecutive evaluation intervals must pass without data for a given dimension before the alert instance is marked as stale and resolved.
If you don't specify a value, Grafana uses the **default of 2 evaluation intervals**.
@@ -25,6 +25,11 @@ refs:
max_attempts:
- pattern: /docs/
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#max_attempts
stale-alert-instances:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/
pending-period:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#pending-period
@@ -109,31 +114,23 @@ The **No Data** state occurs when the alert rule query runs successfully but ret
When an alert instance enters the **No Data** state, Grafana, by default, triggers a new [`DatasourceNoData` alert](#no-data-and-error-alerts). You can control this behavior based on the desired outcome of your alert rule in [Modify the `No Data` or `Error` state](#modify-the-no-data-or-error-state).
### Stale alert instances (MissingSeries)
## Modify the `No Data` or `Error` state
An alert instance is considered **stale** if the query returns data but its dimension (or series) has disappeared for two evaluation intervals.
These states are supported only for Grafana-managed alert rules.
In this case, the alert instance transitions to the **Normal (MissingSeries)** state as resolved, and is then evicted. The process for handling stale alert instances is as follows:
In [Configure no data and error handling](ref:no-data-and-error-handling), you can change the default behavior when the evaluation returns no data or an error. You can set the alert instance state to `Alerting`, `Normal`, `Error`, or `Keep Last State`.
1. The alert rule runs and returns data for some label sets.
{{< figure src="/media/docs/alerting/alert-rule-configure-no-data-and-error-v2.png" alt="A screenshot of the `Configure no data and error handling` option in Grafana Alerting." max-width="500px" >}}
1. An alert instance that previously existed is now missing.
{{< docs/shared lookup="alerts/table-configure-no-data-and-error.md" source="grafana" version="<GRAFANA_VERSION>" >}}
1. Grafana keeps the previous state of the alert instance for two evaluation intervals.
Note that when you configure the **No Data** or **Error** behavior to `Alerting` or `Normal`, Grafana attempts to keep a stable set of fields under notification `Values`. If your query returns no data or an error, Grafana re-uses the latest known set of fields in `Values`, but will use `-1` in place of the measured value.
1. If it remains missing after two intervals, it transitions to the **Normal** state and sets **MissingSeries** in the `grafana_state_reason` annotation.
### Keep last state
1. Stale alert instances in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**, and are routed for notifications like other resolved alerts.
The "Keep Last State" option helps mitigate temporary data source issues, preventing alerts from unintentionally firing, resolving, and re-firing.
1. The alert instance is removed from the UI.
{{< admonition type="tip" >}}
For common examples and practical guidance on handling **Error**, **No Data**, and **stale** alert scenarios, see the following related guides:
- [Handling connectivity errors](ref:guide-connectivity-errors)
- [Handling missing data](ref:guide-missing-data)
{{< /admonition >}}
However, in situations where strict monitoring is critical, relying solely on the "Keep Last State" option may not be appropriate. Instead, consider using an alternative or implementing additional alert rules to ensure that issues with prolonged data source disruptions are detected.
### `No Data` and `Error` alerts
@@ -149,18 +146,6 @@ You can manage these alerts like regular ones by using their labels to apply act
If the alert rule is configured to send notifications directly to a selected contact point (instead of using notification policies), the `DatasourceNoData` and `DatasourceError` alerts are also sent to that contact point. Any additional notification settings defined in the alert rule, such as muting or grouping, are preserved.
## Modify the `No Data` or `Error` state
These states are supported only for Grafana-managed alert rules.
In [Configure no data and error handling](ref:no-data-and-error-handling), you can change the default behavior when the evaluation returns no data or an error. You can set the alert instance state to `Alerting`, `Normal`, `Error`, or `Keep Last State`.
{{< figure src="/media/docs/alerting/alert-rule-configure-no-data-and-error-v2.png" alt="A screenshot of the `Configure no data and error handling` option in Grafana Alerting." max-width="500px" >}}
{{< docs/shared lookup="alerts/table-configure-no-data-and-error.md" source="grafana" version="<GRAFANA_VERSION>" >}}
Note that when you configure the **No Data** or **Error** behavior to `Alerting` or `Normal`, Grafana attempts to keep a stable set of fields under notification `Values`. If your query returns no data or an error, Grafana re-uses the latest known set of fields in `Values`, but will use `-1` in place of the measured value.
### Reduce `No Data` or `Error` alerts
To minimize the number of **No Data** or **Error** state alerts received, try the following.
@@ -174,28 +159,30 @@ To minimize the number of **No Data** or **Error** state alerts received, try th
1. To reduce multiple notifications from **Error** alerts, define a [notification policy](ref:notification-policies) to handle all related alerts with `alertname=DatasourceError`, and filter and group errors from the same data source using the `datasource_uid` label.
### Keep last state
{{< admonition type="tip" >}}
The "Keep Last State" option helps mitigate temporary data source issues, preventing alerts from unintentionally firing, resolving, and re-firing.
For common examples and practical guidance on handling **Error**, **No Data**, and **stale** alert scenarios, see the following related guides:
However, in situations where strict monitoring is critical, relying solely on the "Keep Last State" option may not be appropriate. Instead, consider using an alternative or implementing additional alert rules to ensure that issues with prolonged data source disruptions are detected.
- [Handling connectivity errors](ref:guide-connectivity-errors)
- [Handling missing data](ref:guide-missing-data)
{{< /admonition >}}
## `grafana_state_reason` for troubleshooting
Occasionally, an alert instance may be in a state that isn't immediately clear to everyone. For example:
- [Stale alert instances](#stale-alert-instances-missingseries) in the `Alerting` state transition to the `Normal` state when the series disappear.
- If "no data" handling is configured to transition to a state other than `No Data`.
- If "error" handling is configured to transition to a state other than `Error`.
- If the alert rule is deleted, paused, or updated in some cases, the alert instance also transitions to the `Normal` state.
- [Stale alert instances](ref:stale-alert-instances) in the `Alerting` state transition to the `Normal` state when the series disappear.
In these situations, the evaluation state may differ from the alert state, and it might be necessary to understand the reason for being in that state when receiving the notification.
The `grafana_state_reason` annotation is included in these situations, providing the reason that explains why the alert instance transitioned to its current state. For example:
- [Stale alert instances](#stale-alert-instances-missingseries) in the `Normal` state include the `grafana_state_reason` annotation with the value **MissingSeries**.
- If "no data" or "error" handling transitions to the `Normal` state, the `grafana_state_reason` annotation is included with the value **No Data** or **Error**, respectively.
- If the alert rule is deleted or paused, the `grafana_state_reason` is set to **Paused** or **RuleDeleted**. For some updates, it is set to **Updated**.
- [Stale alert instances](ref:stale-alert-instances) in the `Normal` state include the `grafana_state_reason` annotation with the value **MissingSeries**.
## Alert rule state
+5 -7
View File
@@ -38,9 +38,9 @@ refs:
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/#modify-the-no-data-or-error-state
stale-alert-instances:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/#stale-alert-instances-missingseries
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/#stale-alert-instances-missingseries
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/
no-data-and-error-alerts:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/#no-data-and-error-alerts
@@ -151,9 +151,9 @@ Because of this, `DatasourceNoData` alerts might require a dedicated setup to ha
_MissingSeries_ occurs when only some series disappear but not all. This case is subtle — but important.
Grafana marks missing series as [**stale**](ref:stale-alert-instances) after two evaluation intervals and triggers the alert instance eviction process. Here’s what happens under the hood:
By default, Grafana marks missing series as [**stale**](ref:stale-alert-instances) after two consecutive evaluation intervals without data and triggers the alert instance eviction process. Here’s what happens under the hood:
- Alert instances with missing data keep their last state for two evaluation intervals.
- Alert instances with missing data keep their last state for a number of consecutive evaluation intervals, defined by the **Missing series evaluations to resolve** option (default: 2).
- If still missing after that:
- Grafana adds the annotation `grafana_state_reason: MissingSeries`.
- The alert instance transitions to the `Normal` state.
@@ -171,8 +171,6 @@ If an alert instance becomes stale, you’ll find in the [alert history](ref:ale
| 04:00 | 1.4s 🟢 | — | 🟢 🟢 `region2` Normal (Missing Series), resolved, and instance evicted; 📩 Notification sent. |
| 05:00 | 1.4s 🟢 | — | 🟢 No Alerts |
###
### Why doesn’t MissingSeries match No Data behavior?
In dynamic environments — autoscaling groups, ephemeral pods, spot instances — series naturally come and go. **MissingSeries** normally signals infrastructure or deployment changes.
@@ -204,5 +202,5 @@ Grafana Alerting handles distinct scenarios automatically. Here’s how to think
- Understand `DatasourceNoData` and `MissingSeries` notifications, since they don’t behave like regular alerts.
- Use `absent()` or `absent_over_time()` in Prometheus for fine-grained detection when a metric or label disappears entirely.
- Don’t alert on every instance by default. In dynamic environments, it’s better to aggregate and alert on symptoms — unless a missing individual instance directly impacts users.
- If you’re getting too much noise from disappearing data, consider adjusting alerts, using `Keep Last State`, or routing those alerts differently.
- If you’re getting too much noise from disappearing data, consider adjusting alerts, using `Keep Last State` and the `Missing series evaluations to resolve` option, or routing those alerts differently.
- For connectivity issues involving alert query failures, see the sibling guide: [Handling connectivity errors in Grafana Alerting](ref:connectivity-errors-guide).
@@ -42,14 +42,9 @@ refs:
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/#alert-rule-state
alert-instance-state:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/#alert-instance-state
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/#alert-instance-state
alert-instance-state:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/#alert-instance-state
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/#alert-instance-state
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances/
alert-rule-health:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/