What is this feature?
This PR fixes a state transition issue where alerts transitioning from the Recovering state back to the Alerting state incorrectly entered the Pending state first if the rule had a For duration configured.
Why do we need this feature?
When an alert goes from Alerting to Recovering (when using the Keep firing for) and then back to Alerting, the existing logic would incorrectly put the alert into Pending state while it should be alerting and still sending notifications to the Alertmanager.
* placeholder commit
* Complete function in api_convert_prometheus.go
* MVP before extensive testing
* Cleanup
* Updated tests
* cleanup
* Fix random logs and lint
* Remove comment
* Fix errors after rebase
* Update test
* Update swagger
* swagger
* Refactor to accept groups in body
* Fix auth tests and some cleanup
* Some cleanup before refactoring
* Remove unnecessary fields
* Also refactor RouteConvertPrometheusPostRuleGroup
* Remove unused code
* Rebase + cleanup
* Update authorization_test
* address comments
* Regen swagger files
* Remove namespace and group filters
* Final comments
* remove feature flag alertingAlertmanagerExtraDedupStage
* use most recent version of fork of alertmanager
---------
Co-authored-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
What is this feature?
A follow-up for #101184, adds AlertRule.MissingSeriesEvalsToResolve to the APIs.
missing_series_evals_to_resolve must be specified too and it must be > 0.
POST /api/ruler/grafana/api/v1/rules/{folderUID} works in the following way:
If missing_series_evals_to_resolve is not sent or null, the rule keeps its existing value
If missing_series_evals_to_resolve > 0: updates to that value
If missing_series_evals_to_resolve = 0: resets to default (nil).
AlertRule.MissingSeriesEvalsToResolve can't be 0, so I used it to reset
In the Provisioning API, the value is just set if present and > 0. Otherwise it's reset:
PUT to /api/v1/provisioning/alert-rules/{UID}:
If missing_series_evals_to_resolve is nil, it's reset to the default value
If missing_series_evals_to_resolve > 0, it's updated
What is this feature?
Prometheus conversion: ensures that AlertRule.Data queries always have default parameters set (intervalMs, maxDataPoints). Without this, updates of the same rule can cause version increments.
Why do we need this feature?
Currently, when converting Prometheus rules to Grafana alerts, some default parameters are not explicitly set in the query model. This creates a problem during rule updates:
When a user updates a rule that hasn't changed, we still detect differences in the AlertQuery.Model because the newly converted rules are missing the default fields, such as intervalMs and maxDataPoints. This causes unnecessary version increments of alert rules.
What is this feature?
This PR changes the behavior of the $value and .Value variables in alerting templating to be more compatible with Prometheus templating. When a single datasource is used in the alerting rule, these variables will now return the numeric value from the query instead of the evaluation string.
Why do we need this feature?
It makes Grafana templating more compatible with Prometheus templates. In Prometheus, $value returns the numeric value of the query, but in Grafana it's the evaluation string: [ var='A' labels={instance=instance1} value=81.234 ]. This is because in Grafana multiple datasources can be used in the alert rule, and it's not always possible to get a single value.
This change makes Grafana's behavior consistent with Prometheus when a single datasource is used, and in case when multiple datasources are used in the query, it keeps the old behaviour.
Both $value and .Value are not recommended to use (documentation), and it's better to use .Values instead.
* Add dashboard cleanup job
Change log message
Adjust logic to account for new head RV logic
Don't update lastResourceVersion due to pagination
Save improvements
* Address review feedback
* Update docs.
* Remove docs
* Rename config
---------
Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
Adds support for rule group-level query_offset in Prometheus to Grafana rule conversion. It allows specifying a time offset for rule evaluation, which gets applied and saved during the conversion.
Adds a new configuration option to specify a time offset for rule evaluation, which gets applied and saved during the Prometheus -> Grafana conversion. For example:
[unified_alerting.prometheus_conversion]
rule_query_offset = 1m
Changing this option affects only the rules imported after the change. If query_offset is set at the group level, it takes precedence over this setting.
Default is set to 1m.
* Alerting: Fix loss of TimeInterval location on remote AM apply
deepcopy.Copy does not correctly copy PostableUserConfig because it ignores
unexported fields. As a result, TimeInterval locations default to UTC instead
of retaining their original values.
* make update-workspace
What is this feature?
This PR introduces a new alert rule configuration option, keep_firing_for (Prometheus documentation).
keep_firing_for prevents alerts from resolving immediately after the alert condition returns to normal. Instead, they transition into a "Recovering" state and are not considered resolved by the Alertmanager. Once the recovery period ends (or after the next evaluation if it is bigger than keep_firing_for), the alert transitions to "Normal" if it doesn't start alerting again:
Before
+----------+ +----------+
| Alerting |---->| Normal |
+----------+ +----------+
-----
After
+----------+ +------------+ +----------+
| Alerting |----->| Recovering |---->| Normal |
+----------+ +------------+ +----------+
Why do we need this feature?
This feature prevents flapping alerts by adding a recovery period. This helps avoid false resolutions caused by brief alert
* chore: use 'Grafana IRM' wording in alerting contact point
* revert temp condition change
* remove unneeded ts assertion
* more renaming
* use translations
* update test
* running make i18n-extract
* avoid "simple" word in copy
* add query parameter to existing APIs to control the permanent deletion of rules
* add GUID to gettable rule
* add new endpoint /ruler/grafana/api/v1/trash/rule/guid/{RuleGUID} to delete rules from trash permanently
---------
Signed-off-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
* remove feature flag
* remove feature flag in state manager
* make sure no data with empty results is handled
Signed-off-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
---------
Signed-off-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
Adds HMAC-SHA256 signature support to webhook notifications, providing a way to verify the authenticity and integrity of webhook requests. The implementation allows to specify the header in which the signature will be sent. The signature is calculated from the request body.
An optional timestamp header name can be provided. If set, the HMAC signature will be generated by concatenating the timestamp, a ":" and the request body: {timestamp}:{body}. The timestamp will also be sent in the provided header name.
When creating Grafana-managed alerts from Prometheus rule definitions with mimirtool or cortextool, the rules are marked as "provisioned" and are not editable in the Grafana UI. This PR allows changing this by providing an extra header: --extra-header="X-Disable-Provenance=true".
When provenance is disabled, we do not keep the original rule definition in YAML, so it is impossible to read it back using the Prometheus conversion API (mimirtool/cortextool). This is intentional because if we did keep it and the rule was later changed in the UI, its Prometheus YAML definition would no longer reflect the latest version of the alert rule, as it would be unchanged.
The remote write path differs based on whether the data source is actually
Prometheus, Mimir, Cortex, or an older version of Cortex. We do not want
users to have to specify the path, so this change determines the path as
best it can.
It may be in the future we have to make this configurable per-datasource
to cater for setups where it's impossible to determine the correct path.
Alerting: Include time range in templated dashboard and panel urls
Time range:
from=Alert.StartsAt-1hr
Firing Alerts: to=Current Timestamp
Resolved Alerts: to=Alert.EndsAt
What is this feature?
Adds target datasource UID to the recording rules so that they write to the same datasource used for alerting rule queries after the import.
Why do we need this feature?
Target datasourse support was added in #101678, and under a feature flag grafanaManagedRecordingRulesDatasources (#101778).
This PR makes the importing process:
Check if the import contains recording rules
Verify both recording rules and the grafanaManagedRecordingRulesDatasources feature flag are enabled
If either check fails, return an error
If both checks pass, create recording rules with the provided datasource UID set as both the query and target datasource
* Alerting: Allow selection of recording rule write target on per-rule basis.
Introduces a new feature flag (`grafanaManagedRecordingRulesDatasources`),
disabled by default, to enable the ability to write recording rules data using
data source settings, and selecting the data source to use on a per-rule basis.
To cope with the scenario of users upgrading, a configuration file option
allows setting the default data source to use, if none is specified in the rule,
emulating the behaviour of recording rules without the flag enabled.
* Lint
* Update conf/sample.ini
Co-authored-by: Alexander Akhmetov <me@alx.cx>
---------
Co-authored-by: Alexander Akhmetov <me@alx.cx>
Alexander Akhmetov
William Wernert
Will Assis
Will Browne
Yuri Tseretyan
maicon
Fayzal Ghantiwala
Mariell Hoversholm
alerting-team[bot]
Peter Štibraný
Steve Simpson
Marco de Abreu
Matthew Jacobson
Tito Lins
Dominik Broj