public/grafana
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Alerting: Resend alerts for states that are missing in the eval results (#105965)

Browse Source 
What is this feature?

This PR fixes the MissingSeriesEvalsToResolve behavior when it's set to more than 4 evaluation intervals.

Why do we need this feature?

The MissingSeriesEvalsToResolve setting was not working correctly due to alerts being auto-resolved by Alertmanager after 4 evaluation intervals (via the endsAt field).

Before we had deleteStaleStatesFromCache method that was returning only stale states that had to be resolved. Non-stale states for which the current evaluation does not have a series never had endsAt updated and were never resend to the Alertmanager, so they were automatically resolved after 4 evaluations regardless of the setting.

The new processMissingSeriesStates returns state for each missing series on every evaluation, and resolves the stale ones. This guarantees that alerts without series still alert for the configured number of evaluations.
This commit is contained in:
Alexander Akhmetov
2025-05-29 23:22:35 +02:00
committed by GitHub
parent 0a104cc0ed
commit eae77aa695
6 changed files with 363 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pkg/services/ngalert/state/cache.go
+5 -7
View File
@@ -167,25 +167,23 @@ func expand(ctx context.Context, log log.Logger, name string, original map[strin
return expanded, errs
}
func (rs *ruleStates) deleteStates(predicate func(s *State) bool) []*State {
deleted := make([]*State, 0)
func (rs *ruleStates) deleteStates(predicate func(s *State) bool) {
for id, state := range rs.states {
if predicate(state) {
delete(rs.states, id)
deleted = append(deleted, state)
}
}
return deleted
}
func (c *cache) deleteRuleStates(ruleKey ngModels.AlertRuleKey, predicate func(s *State) bool) []*State {
// deleteRuleStates iterates over all states for the given rule and deletes those where predicate returns true.
// The predicate function is called once for each state and should return true if the state should be deleted.
func (c *cache) deleteRuleStates(ruleKey ngModels.AlertRuleKey, predicate func(s *State) bool) {
c.mtxStates.Lock()
defer c.mtxStates.Unlock()
ruleStates, ok := c.states[ruleKey.OrgID][ruleKey.UID]
if ok {
return ruleStates.deleteStates(predicate)
ruleStates.deleteStates(predicate)
}
return nil
}
func (c *cache) setRuleStates(ruleKey ngModels.AlertRuleKey, s ruleStates) {