What is this feature?
Ensures that resolved notifications are sent when alert states transition from Error to Normal after the configured number of evaluation intervals: Missing series evaluations to resolve.
Why do we need this feature?
Before this change, when an alert was transitioning from Error to Normal, in case when the labels on the new Normal alert instance are the same, Grafana would not send resolved notifications for the Error alert state. The alert would be resolved after a few evaluation intervals automatically in the alertmanager, following the endsAt.
With this change the resolved notification is sent after the configured number of evaluation intervals: Missing series evaluations to resolve.
What is this feature?
Fixes a bug when group-level query_offset and labels parameters are ignored and not saved
Why do we need this feature?
In the import API Prometheus YAML rule definitions are supported:
groups:
- name: group-1
interval: 1m
query_offset: 10m
labels:
severity: "warning"
rules:
- alert: Alert 0 > 0
expr: vector(0) > 0
But applying group-level labels and query_offset is broken and they are not saved right now because during the conversion of the API model to PrometheusRuleGroup they aren't saved to the new structure.
* Alerting: Optimize prometheus api permission checks
This improves the performance of the Prometheus API by performing the permission checks for rule read permission in a folder upfront, rather than checking permissions for each rule group individually. This reduces the number of permission checks and should speed up the API response time.
* refactor vars
---------
Co-authored-by: Konrad Lalik <konradlalik@gmail.com>
When a rule configured with `ExecErrState` state of `Alerting`, has an instance which is Alerting then has a data source error, then successfully evaluates and continues to be Alerting, the cached instance keeps the error cached until it is no longer firing.
This is unexpected and leads to misleading results.
What is this feature?
This PR fixes the MissingSeriesEvalsToResolve behavior when it's set to more than 4 evaluation intervals.
Why do we need this feature?
The MissingSeriesEvalsToResolve setting was not working correctly due to alerts being auto-resolved by Alertmanager after 4 evaluation intervals (via the endsAt field).
Before we had deleteStaleStatesFromCache method that was returning only stale states that had to be resolved. Non-stale states for which the current evaluation does not have a series never had endsAt updated and were never resend to the Alertmanager, so they were automatically resolved after 4 evaluations regardless of the setting.
The new processMissingSeriesStates returns state for each missing series on every evaluation, and resolves the stale ones. This guarantees that alerts without series still alert for the configured number of evaluations.
* replace usage of folder guardians with access control evaluators
* remove NewByFolderUID guardian
* bring up to date
* fix test
* more test fixes, and don't fetch the folder before evaluating lib element access
* change what error is returned
* fix alerting test
* try to fix linter errors
* replace the use of newByFolder guardian with direct access control evaluator checks
* remove newByFolder guardian
* get rid of dashboard and folder guardians
* undo unwanted change
* undo unwanted change
* undo unwanted change
* update code owners
* replace usage of folder guardians with access control evaluators
* remove NewByFolderUID guardian
* bring up to date
* fix test
* more test fixes, and don't fetch the folder before evaluating lib element access
* change what error is returned
* fix alerting test
* try to fix linter errors
* we don't assign general folder write permissions, reverting to the previous logic for general folders
* Dependencies: Bump github.com/openfga/openfga from v1.8.6 to v1.8.12
* Linter: Replace x/exp/rand with math/rand/v2
* NGAlert: Fix test after linter fixes
Adds ability to set notifications settings using the Prometheus conversion API.
The API now supports a new optional header: X-Grafana-Alerting-Notification-Settings which can be used to specify notification settings.
The value of the header is the AlertRuleNotificationSettings structure in JSON:
mimirtool rules load alerts.yaml --extra-headers 'X-Grafana-Alerting-Notification-Settings: {"receiver": "my-webhook", "group_by": ["cluster", "pod"]}'
* Update grafana alerting from de176b4a0309 to 83b6de6b0a35
Includes:
- https://github.com/grafana/alerting/pull/319
- https://github.com/grafana/alerting/pull/317
* Remove unused SendWebhook method from sender struct
grafana/alerting hasn't used the grafana webhook sender for a while now,
so this method is no longer used anywhere.
- Removed SendWebhook from the sender struct and rename it to emailSender
so that its use is clearer.
- Also, for similar reasons, the Webhook method on Grafana's
webhook sender `sendWebRequestSync` should not call grafana/alerting code for
NewTLSClient. The previous grafana/alerting function is vendored into grafana.
* Use BuildReceiverIntegrations new func signature
Fayzal Ghantiwala
Alexander Akhmetov
Tito Lins
Vadim Stepanov
Moustafa BaiouandKonrad Lalik
Yuri Tseretyan
Peter Štibraný
William Wernert
Matheus Macabu
Santiago
Ieva
beejeebus
Matthew Jacobson
Sonia AguilarandGilles De Mey