Files
grafana/pkg/services/ngalert/api/tooling
Alexander Akhmetov 695ac91290 Alerting: Add backend support for keep_firing_for (#100750)
What is this feature?

This PR introduces a new alert rule configuration option, keep_firing_for (Prometheus documentation).

keep_firing_for prevents alerts from resolving immediately after the alert condition returns to normal. Instead, they transition into a "Recovering" state and are not considered resolved by the Alertmanager. Once the recovery period ends (or after the next evaluation if it is bigger than keep_firing_for), the alert transitions to "Normal" if it doesn't start alerting again:

Before                                          

+----------+     +----------+                    
| Alerting |---->|  Normal  |                    
+----------+     +----------+                    

-----
After

+----------+      +------------+     +----------+
| Alerting |----->| Recovering |---->|  Normal  |
+----------+      +------------+     +----------+                                                 

Why do we need this feature?

This feature prevents flapping alerts by adding a recovery period. This helps avoid false resolutions caused by brief alert
2025-03-18 11:24:48 +01:00
..
2024-02-26 10:52:44 -07:00

What

This aims to define the unified alerting API as code. It generates OpenAPI definitions from go structs. It also generates server/route stubs based on our documentation.

Running

make - regenerate everything - documentation and server stubs. make serve - regenerate the Swagger document, and host rendered docs on port 80. view api

Requires

Why

The current state of Swagger extraction from golang is relatively limited. It's easier to generate server stubs from an existing Swagger doc, as there are limitations with producing a Swagger doc from a hand-written API stub. The current extractor instead relies on comments describing the routes, but the comments and actual implementation may drift, which we don't want to allow.

Instead, we use a hybrid approach - we define the types in Golang, with comments describing the routes, in a standalone package with minimal dependencies. From this, we produce a Swagger doc, and then turn the Swagger doc back into a full-blown server stub.

Stability

We have some endpoints that we document publicly as being stable, and others that we consider unstable. The stable endpoints are documented in api.json, where all endpoints are available in post.json.

To stabilize an endpoint, add the stable tag to its route comment:

// swagger:route GET /provisioning/contact-points provisioning stable RouteGetContactpoints