Alerting: Add backend support for keep_firing_for (#100750)

What is this feature?

This PR introduces a new alert rule configuration option, keep_firing_for (Prometheus documentation).

keep_firing_for prevents alerts from resolving immediately after the alert condition returns to normal. Instead, they transition into a "Recovering" state and are not considered resolved by the Alertmanager. Once the recovery period ends (or after the next evaluation if it is bigger than keep_firing_for), the alert transitions to "Normal" if it doesn't start alerting again:

Before                                          

+----------+     +----------+                    
| Alerting |---->|  Normal  |                    
+----------+     +----------+                    

-----
After

+----------+      +------------+     +----------+
| Alerting |----->| Recovering |---->|  Normal  |
+----------+      +------------+     +----------+                                                 

Why do we need this feature?

This feature prevents flapping alerts by adding a recovery period. This helps avoid false resolutions caused by brief alert

pkg/services/ngalert/models/instance.go

@@ -39,6 +39,8 @@ const (
 	InstanceStateNoData InstanceStateType = "NoData"
 	// InstanceStateError is for an erroring alert.
 	InstanceStateError InstanceStateType = "Error"
+	// InstanceStateRecovering is for a recovering alert.
+	InstanceStateRecovering InstanceStateType = "Recovering"
 )
 
 // IsValid checks that the value of InstanceStateType is a valid
@@ -48,7 +50,8 @@ func (i InstanceStateType) IsValid() bool {
 		i == InstanceStateNormal ||
 		i == InstanceStateNoData ||
 		i == InstanceStatePending ||
-		i == InstanceStateError
+		i == InstanceStateError ||
+		i == InstanceStateRecovering
 }
 
 // ListAlertInstancesQuery is the query list alert Instances.