public/grafana
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[release-12.0.3] docs(alerting): Add Detect missing series in Prometheus section to the MissingData guide (#107374)

Browse Source 
* docs(alerting): Add `Detect missing series in Prometheus` section to the MissingData guide (#107329)

(cherry picked from commit 2f1a6ae171)

* fix ref link
This commit is contained in:
Pepe Cano
2025-06-30 17:32:11 +02:00
committed by GitHub
parent e5cfceac59
commit b2825ffde9
2 changed files with 59 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sources/alerting/best-practices/missing-data.md
+53 -9
View File
@@ -46,6 +46,11 @@ refs:
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/#no-data-and-error-alerts
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/#no-data-and-error-alerts
grafana-state-reason-annotation:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/#grafana_state_reason-for-troubleshooting
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/#grafana_state_reason-for-troubleshooting
---
# Handle missing data in Grafana Alerting
@@ -114,12 +119,18 @@ However, `absent_over_time()` can’t detect which specific series are missing s
If you want to check for missing data per-region or label, you can specify the label in the alert query as follows:
`absent_over_time(http_request_latency_seconds{region="region1"}[5m]) == 1`
`or`
`absent_over_time(http_request_latency_seconds{region="region2"}[5m]) == 1`
```promQL
# Detect missing data in region1
absent_over_time(http_request_latency_seconds{region="region1"}[5m]) == 1
# Detect missing data in region2
absent_over_time(http_request_latency_seconds{region="region2"}[5m]) == 1
```
But this doesn't scale well. It is unreliable to have hard-coded queries for each label set, especially in dynamic cloud environments where instances can appear or disappear at any time.
To detect when a specific target has disappeared, see below **Evict alert instances for missing series** for details on how Grafana handles this case and how to set up detection.
## Manage No Data issues in Grafana alerts
While Prometheus provides functions like `absent_over_time()` to detect missing data, not all data sources — like Graphite, InfluxDB, PostgreSQL, and others — available to Grafana alerts support a similar function.
@@ -168,8 +179,6 @@ If an alert instance becomes stale, you’ll find it in the [alert history](ref:
| 03:00 | 1.4s 🟢 | `(MissingSeries)` <br> `Normal` | 🟢🟢 `region2` was resolved, 📩 notification sent, and instance evicted. |
| 04:00 | 1.4s 🟢 | — | 🟢 No Alerts. `region2` was evicted. |
###
### Why doesn’t MissingSeries match No Data behavior?
In dynamic environments, such as autoscaling groups, ephemeral pods, spot instances, series naturally come and go. **MissingSeries** normally signals infrastructure or deployment changes.
@@ -182,15 +191,50 @@ In environments with frequent scale events, prioritize symptom-based alerts over
### Handle MissingSeries notifications
A stale alert instance triggers a **resolved notification** if it transitions from a firing state (such as `Alerting`, `No Data`, or `Error`) to `Normal`.
A stale alert instance triggers a **resolved notification** if it transitions from a firing state (such as `Alerting`, `No Data`, or `Error`) to `Normal`, and the [`grafana_state_reason` annotation](ref:grafana-state-reason-annotation) is set to **MissingSeries** to indicate that the alert wasn’t resolved by recovery but evicted because the series data went missing.
You can display the `MissingSeries` annotation in notifications to indicate the alert wasn’t resolved by recovery but evicted due to series data going missing.
Recognizing these notifications helps you handle them appropriately. For example:
Review these notifications to confirm whether something broke or if the alert was unnecessary. To reduce noise:
- Display the `grafana_state_reason` annotation to clearly identify **MissingSeries** alerts.
- Or use the `grafana_state_reason` annotation to process these alerts differently.
Also, review these notifications to confirm whether something broke or if the alert was unnecessary. To reduce noise:
- Silence or mute alerts during planned maintenance or rollouts.
- Adjust alert rules to avoid triggering on series you expect to come and go, and use aggregated alerts instead.
### Detect missing series in Prometheus
Previously, an example showed how to detect missing data for a specific label, such as `region`:
```promQL
# Detect missing data in region1
absent_over_time(http_request_latency_seconds{region="region1"}[5m]) == 1
# Detect missing data in region2
absent_over_time(http_request_latency_seconds{region="region2"}[5m]) == 1
```
However, this approach doesn’t scale well because it requires hardcoding all possible `region` values.
As an alternative, you can create an alert rule that detects missing series dynamically using the `present_over_time` function:
```promQL
present_over_time(http_request_latency_seconds{}[24h])
unless
present_over_time(http_request_latency_seconds{}[10m])
```
Or, if you want to group by a label such as region:
```promQL
group(present_over_time(http_request_latency_seconds{}[24h])) by (region)
unless
group(present_over_time(http_request_latency_seconds{}[10m])) by (region)
```
This query finds regions (or other targets) that were present at any time in the past 24 hours but have not been present in the past 10 minutes. The alert rule then triggers an alert instance for each missing region. You can apply the same technique to any label or target dimension.
## Conclusion
Missing data isn’t always a failure. It’s a common scenario in dynamic environments when certain targets stop reporting.
@@ -200,7 +244,7 @@ Grafana Alerting handles distinct scenarios automatically. Here’s how to think
- Understand `DatasourceNoData` and `MissingSeries` notifications, since they don’t behave like regular alerts.
- Use Grafana’s _No Data_ handling options to define what happens when a query returns nothing.
- When _NoData_ is not an issue, consider rewriting the query to always return data — for example, in Prometheus, use `your_metric_query OR on() vector(0)` to return `0` when `your_metric_query` returns nothing.
- Use `absent()` or `absent_over_time()` in Prometheus for fine-grained detection when a metric or label disappears entirely.
- Use `absent_over_time()` or `present_over_time` in Prometheus to detect when a metric or target disappears.
- If data is frequently missing due to scrape delays, use techniques to account for data delays:
- Adjust the **Time Range** query option in Grafana to evaluate slightly behind real time (e.g., set **To** to `now-1m`) to account for late data points.
- In Prometheus, you can use `last_over_time(metric_name[10m])` to pick the most recent sample within a given window.
docs/sources/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances.md
+6 -1
View File
@@ -29,6 +29,11 @@ refs:
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/missing-data/
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/best-practices/missing-data/
grafana-state-reason-annotation:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/state-and-health/#grafana_state_reason-for-troubleshooting
- pattern: /docs/grafana-cloud/
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rule-evaluation/state-and-health/#grafana_state_reason-for-troubleshooting
---
# Stale alert instances
@@ -60,7 +65,7 @@ The process for handling stale alert instances is as follows:
1. Grafana keeps the previous state of the alert instance for the number of evaluation intervals specified in [Missing series evaluations to resolve](#configure-missing-series-evaluations-to-resolve).
1. If it remains missing after the specified number of evaluation intervals (2 by default), it transitions to the **Normal** state and sets **MissingSeries** in the `grafana_state_reason` annotation.
1. If it remains missing after the specified number of evaluation intervals (2 by default), it transitions to the **Normal** state and sets **MissingSeries** in the [`grafana_state_reason` annotation](ref:grafana-state-reason-annotation).
Stale alert instances in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**, and are routed for notifications like other resolved alerts.