From be8d153dc33734caba4f617ff571d18253e68fa0 Mon Sep 17 00:00:00 2001 From: Pepe Cano <825430+ppcano@users.noreply.github.com> Date: Thu, 5 Jun 2025 00:05:22 +0200 Subject: [PATCH] docs(alerting): updates Missing series evaluations to resolve option (#106319) docs(alerting): correct detailed process of the missingSeriesToEval option --- .../alerting/best-practices/missing-data.md | 21 ++++++++----------- .../stale-alert-instances.md | 12 +++++------ 2 files changed, 15 insertions(+), 18 deletions(-) diff --git a/docs/sources/alerting/best-practices/missing-data.md b/docs/sources/alerting/best-practices/missing-data.md index b4997452cb2..ac527d305b8 100644 --- a/docs/sources/alerting/best-practices/missing-data.md +++ b/docs/sources/alerting/best-practices/missing-data.md @@ -88,8 +88,7 @@ The following tables illustrate both scenarios using the previous example, with | :---- | :--------- | :--------- | :--------------------------- | | 00:00 | 1.5s 🟒 | 1s 🟒 | βœ… No Alert | | 01:00 | No Data ⚠️ | No Data ⚠️ | ⚠️ No Alert (Silent Failure) | -| 02:00 | No Data ⚠️ | No Data ⚠️ | ⚠️ No Alert (Silent Failure) | -| 03:00 | 1.4s 🟒 | 1s 🟒 | βœ… No Alert | +| 02:00 | 1.4s 🟒 | 1s 🟒 | βœ… No Alert | **MissingSeries Scenario:** Only a specific series (`region2`) disappears: @@ -97,8 +96,7 @@ The following tables illustrate both scenarios using the previous example, with | :---- | :------ | :---------------- | :--------------------------- | | 00:00 | 1.5s 🟒 | 1s 🟒 | βœ… No Alert | | 01:00 | 1.6s 🟒 | Missing Series ⚠️ | ⚠️ No Alert (Silent Failure) | -| 02:00 | 1.6s 🟒 | Missing Series ⚠️ | ⚠️ No Alert (Silent Failure) | -| 03:00 | 1.4s 🟒 | 1s 🟒 | βœ… No Alert | +| 02:00 | 1.4s 🟒 | 1s 🟒 | βœ… No Alert | In both cases, something broke silently. @@ -162,14 +160,13 @@ Grafana marks missing series as [**stale**](ref:stale-alert-instances) after two If an alert instance becomes stale, you’ll find it in the [alert history](ref:alert-history) as `Normal (Missing Series)` before it disappears. This table shows the eviction process from the previous example: -| Time | region1 | region2 | Alert triggered | -| :---- | :-------------------- | :--------------------------------- | :--------------------------------------------------------------------------------------------- | -| 00:00 | 1.5s 🟒 | 1s 🟒 | 🟒🟒 No Alerts | -| 01:00 | 3s πŸ”΄
`Alerting` | 3s πŸ”΄
`Alerting` | πŸ”΄πŸ”΄ Alert instances triggered for both regions | -| 02:00 | 1.6s 🟒 | MissingSeries ⚠️
`Alerting` ️ | πŸŸ’πŸ”΄ Region2 missing, state maintained. | -| 03:00 | 1.6s 🟒 | MissingSeries ⚠️ `Alerting`️ | πŸŸ’πŸ”΄Region2 missing, state maintained. | -| 04:00 | 1.4s 🟒 | β€” | 🟒 🟒 `region2` Normal (Missing Series), resolved, and instance evicted; πŸ“© Notification sent. | -| 05:00 | 1.4s 🟒 | β€” | 🟒 No Alerts | +| Time | region1 | region2 | Alert triggered | +| :---- | :-------------------- | :------------------------------------ | :----------------------------------------------------------------------- | +| 00:00 | 1.5s 🟒 | 1s 🟒 | 🟒🟒 No Alerts | +| 01:00 | 3s πŸ”΄
`Alerting` | 3s πŸ”΄
`Alerting` | πŸ”΄πŸ”΄ Alert instances triggered for both regions | +| 02:00 | 1.6s 🟒 | `(MissingSeries)`⚠️
`Alerting` ️ | πŸŸ’πŸ”΄ Region2 missing, state maintained. | +| 03:00 | 1.4s 🟒 | `(MissingSeries)`
`Normal` | 🟒🟒 `region2` was resolved, πŸ“© notification sent, and instance evicted. | +| 04:00 | 1.4s 🟒 | β€” | 🟒 No Alerts. `region2` was evicted. | ### diff --git a/docs/sources/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances.md b/docs/sources/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances.md index 8b8227c7027..50983640285 100644 --- a/docs/sources/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances.md +++ b/docs/sources/alerting/fundamentals/alert-rule-evaluation/stale-alert-instances.md @@ -39,10 +39,10 @@ This is different from the [**No Data** state](ref:no-data-state), which occurs A stale alert instance transitions to the **Normal (MissingSeries)** state as **Resolved**, and is then evicted: -| Eval. Interval | 1 | 2 | 3 | 4 | 5 | -| :--------------- | :-- | :-------------- | :-------------- | :-- | :-- | -| Alert instance A | βœ” | βœ” | βœ” | βœ” | βœ” | -| Alert instance B | βœ” | `MissingSeries` | `MissingSeries` | οΈπŸ“© | | +| Eval. Interval | 1 | 2 | 3 | 4 | +| :--------------- | :-- | :-------------- | :--------------------------------------- | :-- | +| Alert instance A | βœ” | βœ” | βœ” | βœ” | +| Alert instance B | βœ” | `MissingSeries` | ️`Normal(MissingSeries)` πŸ“©\* | | {{< admonition type="note" >}} @@ -60,9 +60,9 @@ The process for handling stale alert instances is as follows: 1. Grafana keeps the previous state of the alert instance for the number of evaluation intervals specified in [Missing series evaluations to resolve](#configure-missing-series-evaluations-to-resolve). -1. If it remains missing after two intervals, it transitions to the **Normal** state and sets **MissingSeries** in the `grafana_state_reason` annotation. +1. If it remains missing after the specified number of evaluation intervals (2 by default), it transitions to the **Normal** state and sets **MissingSeries** in the `grafana_state_reason` annotation. -1. Stale alert instances in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**, and are routed for notifications like other resolved alerts. + Stale alert instances in the **Alerting**, **No Data**, or **Error** states transition to the **Normal** state as **Resolved**, and are routed for notifications like other resolved alerts. 1. The alert instance is removed from the UI.