What is this feature?
Prometheus conversion: ensures that AlertRule.Data queries always have default parameters set (intervalMs, maxDataPoints). Without this, updates of the same rule can cause version increments.
Why do we need this feature?
Currently, when converting Prometheus rules to Grafana alerts, some default parameters are not explicitly set in the query model. This creates a problem during rule updates:
When a user updates a rule that hasn't changed, we still detect differences in the AlertQuery.Model because the newly converted rules are missing the default fields, such as intervalMs and maxDataPoints. This causes unnecessary version increments of alert rules.
What is this feature?
This PR changes the behavior of the $value and .Value variables in alerting templating to be more compatible with Prometheus templating. When a single datasource is used in the alerting rule, these variables will now return the numeric value from the query instead of the evaluation string.
Why do we need this feature?
It makes Grafana templating more compatible with Prometheus templates. In Prometheus, $value returns the numeric value of the query, but in Grafana it's the evaluation string: [ var='A' labels={instance=instance1} value=81.234 ]. This is because in Grafana multiple datasources can be used in the alert rule, and it's not always possible to get a single value.
This change makes Grafana's behavior consistent with Prometheus when a single datasource is used, and in case when multiple datasources are used in the query, it keeps the old behaviour.
Both $value and .Value are not recommended to use (documentation), and it's better to use .Values instead.
* Add dashboard cleanup job
Change log message
Adjust logic to account for new head RV logic
Don't update lastResourceVersion due to pagination
Save improvements
* Address review feedback
* Update docs.
* Remove docs
* Rename config
---------
Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
Adds support for rule group-level query_offset in Prometheus to Grafana rule conversion. It allows specifying a time offset for rule evaluation, which gets applied and saved during the conversion.
Adds a new configuration option to specify a time offset for rule evaluation, which gets applied and saved during the Prometheus -> Grafana conversion. For example:
[unified_alerting.prometheus_conversion]
rule_query_offset = 1m
Changing this option affects only the rules imported after the change. If query_offset is set at the group level, it takes precedence over this setting.
Default is set to 1m.
* Alerting: Fix loss of TimeInterval location on remote AM apply
deepcopy.Copy does not correctly copy PostableUserConfig because it ignores
unexported fields. As a result, TimeInterval locations default to UTC instead
of retaining their original values.
* make update-workspace
What is this feature?
This PR introduces a new alert rule configuration option, keep_firing_for (Prometheus documentation).
keep_firing_for prevents alerts from resolving immediately after the alert condition returns to normal. Instead, they transition into a "Recovering" state and are not considered resolved by the Alertmanager. Once the recovery period ends (or after the next evaluation if it is bigger than keep_firing_for), the alert transitions to "Normal" if it doesn't start alerting again:
Before
+----------+ +----------+
| Alerting |---->| Normal |
+----------+ +----------+
-----
After
+----------+ +------------+ +----------+
| Alerting |----->| Recovering |---->| Normal |
+----------+ +------------+ +----------+
Why do we need this feature?
This feature prevents flapping alerts by adding a recovery period. This helps avoid false resolutions caused by brief alert
* chore: use 'Grafana IRM' wording in alerting contact point
* revert temp condition change
* remove unneeded ts assertion
* more renaming
* use translations
* update test
* running make i18n-extract
* avoid "simple" word in copy
* add query parameter to existing APIs to control the permanent deletion of rules
* add GUID to gettable rule
* add new endpoint /ruler/grafana/api/v1/trash/rule/guid/{RuleGUID} to delete rules from trash permanently
---------
Signed-off-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
* remove feature flag
* remove feature flag in state manager
* make sure no data with empty results is handled
Signed-off-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
---------
Signed-off-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
Adds HMAC-SHA256 signature support to webhook notifications, providing a way to verify the authenticity and integrity of webhook requests. The implementation allows to specify the header in which the signature will be sent. The signature is calculated from the request body.
An optional timestamp header name can be provided. If set, the HMAC signature will be generated by concatenating the timestamp, a ":" and the request body: {timestamp}:{body}. The timestamp will also be sent in the provided header name.
When creating Grafana-managed alerts from Prometheus rule definitions with mimirtool or cortextool, the rules are marked as "provisioned" and are not editable in the Grafana UI. This PR allows changing this by providing an extra header: --extra-header="X-Disable-Provenance=true".
When provenance is disabled, we do not keep the original rule definition in YAML, so it is impossible to read it back using the Prometheus conversion API (mimirtool/cortextool). This is intentional because if we did keep it and the rule was later changed in the UI, its Prometheus YAML definition would no longer reflect the latest version of the alert rule, as it would be unchanged.
The remote write path differs based on whether the data source is actually
Prometheus, Mimir, Cortex, or an older version of Cortex. We do not want
users to have to specify the path, so this change determines the path as
best it can.
It may be in the future we have to make this configurable per-datasource
to cater for setups where it's impossible to determine the correct path.
Alerting: Include time range in templated dashboard and panel urls
Time range:
from=Alert.StartsAt-1hr
Firing Alerts: to=Current Timestamp
Resolved Alerts: to=Alert.EndsAt
What is this feature?
Adds target datasource UID to the recording rules so that they write to the same datasource used for alerting rule queries after the import.
Why do we need this feature?
Target datasourse support was added in #101678, and under a feature flag grafanaManagedRecordingRulesDatasources (#101778).
This PR makes the importing process:
Check if the import contains recording rules
Verify both recording rules and the grafanaManagedRecordingRulesDatasources feature flag are enabled
If either check fails, return an error
If both checks pass, create recording rules with the provided datasource UID set as both the query and target datasource
* Alerting: Allow selection of recording rule write target on per-rule basis.
Introduces a new feature flag (`grafanaManagedRecordingRulesDatasources`),
disabled by default, to enable the ability to write recording rules data using
data source settings, and selecting the data source to use on a per-rule basis.
To cope with the scenario of users upgrading, a configuration file option
allows setting the default data source to use, if none is specified in the rule,
emulating the behaviour of recording rules without the flag enabled.
* Lint
* Update conf/sample.ini
Co-authored-by: Alexander Akhmetov <me@alx.cx>
---------
Co-authored-by: Alexander Akhmetov <me@alx.cx>
The change to use WriteDatasource was done in a previous commit, this adds a
test case using DatasourceWriter, in addition to the one using PrometheusWriter.
* Alerting: Implement arbitrary data source remote writer.
Recording rules currently remote write to the endpoint speficied in the
configuration file, but we want to change this so that it can be specified
on a per-rule basis. This change implements the updated RemoteWriter
interface, and dispatches writes to the correct data source.
Notes about the implementation:
- There is a defaulting behaviour, because we will need to handle situations
when upgrading from older Grafana versions, where recording rules do not a
data source target specified.
- In order to avoid re-creating the remote write clients for every request,
a cache is used. The entries have a TTL of 30 seconds at which point they
are recreated; this is the delay for picking up new data source settings.
Known limitations:
- There is no per data source configuration for the remote write path suffix.
- Not all data source settings will be used, but basic auth and TLS will be.
- There is no RBAC. This will be a follow up feature.
* Lint
* Alerting: Refactor NewPrometheusWriter function.
In order to re-use PrometheusWriter, changing the function take a
PrometheusWriterConfig instead of RecordingRulesSettings, and adapt the old
interface onto the new interface.
* Make linter happy
Extend the recording rule definition to include the target data source, allowing
configuration of where the output of the recording rule is written to. Also
extends the relevant interfaces in preparation for the next set of changes.
* add feature toggle alertRuleRestore
* Update delete rule to require UserUID, remove all versions and create "delete" version that holds information about who and when deleted the rule
What is this feature?
Allows the creation of alert rules with mimirtool in a specified folder.
Why do we need this feature?
Currently, the APIs for mimirtool create namespaces and rule groups in the root folder without the ability to set a custom folder. For example, it could be a special "Imported" folder, etc.
This PR makes it possible with a special header: mimirtool ... --extra-headers="X-Grafana-Alerting-Folder-UID=123". If it's not present, the root folder is used, otherwise, the specified one is used.
mimirtool does not support nested folder structures, while Grafana allows folder nesting. To keep compatibility, we return only direct child folders of the working folder (as namespaces) with rule groups and rules that are directly in these child folders as if there are no nested folders.
For example, given this folder structure in Grafana:
```
grafana/
├── production/
│ ├── service1/
│ │ └── alerts/
│ └── service2/
└── testing/
└── service3/
```
If the working folder is "grafana":
Only namespaces "production" and "testing" are returned
Only rule groups directly within these folders are included
If the working folder is "production":
- Only namespaces "service1" and "service2" are returned
Only rule groups directly within these folders are included
Expand template testing to try additional scopes if the root scope fails.
This mitigates errors for definitions like pagerduty.default.instances,
which require the .Alerts scope. Added support for .Alerts and .Alert
scopes.
* add column guid to alert rule table and rule_guid to rule version table
+ populate the new field with UUID
* update storage and domain models
* patch GUID
* ignore GUID in fingerprint tests
Initially, Metadata had only the EditorSettings, and HasMetadata was used to understand if the incoming update request had metadata in the body because it could be omitted if it was empty. For example, when the rule is updated via the provisioning API or has only false values. If it was in the request, we used that; if not, we used the metadata from the existing rule from the database. If the rule was updated via the AlertRuleService, we didn't change Metadata at all if the rule already existed.
But now, Metadata also has the Prometheus rule definition, and we always need to update it with the new version of the AlertRuleService when the rule exists in the DB and has the same UID. HasMetadata is renamed to HasEditorSettings to keep the old behaviour only for EditorSettings.
Now, the provisioning API and the conversion API will overwrite everything except EditorSettings with the new data.
When you use a math expression with out any operators, the dataFrame pointer is identical between the expression result and the input query/expression.
This was resulting in the values returned from an evaluation overshadowing each other, depending on the order of the processing of the result map.
For example:
```
A: some_metric
B: reduce of A
C: math expression -> "${B}"
D: Threshold evaluation of C -> "C > 0"
```
With a value of 1 for `some_metric`, might result in a evaluation result of one of the following (somewhat at random):
1. { B: 1, D: 1 }
2. { C: 1, D: 1}
While you would expect to see:
{ B: 1, C: 1, D: 1 }
Alexander Akhmetov
Peter Štibraný
Steve Simpson
Marco de AbreuandMarco de Abreu
Matthew Jacobson
Yuri Tseretyan
Tito Lins
Dominik Broj
Ryan McKinley
Moustafa Baiou