Recording rule fields were not being copied correctly when duplicating an alert rule. This manifests as missing `TargetDataSourceUID` fields from the `Record` part of the rule when rules in a group are re-ordered.
Added some additional tests to ensure we cover the generation of recording rules in tests and fixed the copying logic to ensure all fields are copied correctly.
(cherry picked from commit c73b3ccf6e)
* Alerting: Resend alerts for states that are missing in the eval results (#105965)
What is this feature?
This PR fixes the MissingSeriesEvalsToResolve behavior when it's set to more than 4 evaluation intervals.
Why do we need this feature?
The MissingSeriesEvalsToResolve setting was not working correctly due to alerts being auto-resolved by Alertmanager after 4 evaluation intervals (via the endsAt field).
Before we had deleteStaleStatesFromCache method that was returning only stale states that had to be resolved. Non-stale states for which the current evaluation does not have a series never had endsAt updated and were never resend to the Alertmanager, so they were automatically resolved after 4 evaluations regardless of the setting.
The new processMissingSeriesStates returns state for each missing series on every evaluation, and resolves the stale ones. This guarantees that alerts without series still alert for the configured number of evaluations.
* Remove FiredAt field
Alerting: Provisioning API returns 403 on quota exceeded for rule group PUT (#106409)
(cherry picked from commit 1df888c517)
Co-authored-by: Vadim Stepanov <vstpme@gmail.com>
Alerting: Fix group-level labels and query_offset in the import API (#106379)
What is this feature?
Fixes a bug when group-level query_offset and labels parameters are ignored and not saved
Why do we need this feature?
In the import API Prometheus YAML rule definitions are supported:
groups:
- name: group-1
interval: 1m
query_offset: 10m
labels:
severity: "warning"
rules:
- alert: Alert 0 > 0
expr: vector(0) > 0
But applying group-level labels and query_offset is broken and they are not saved right now because during the conversion of the API model to PrometheusRuleGroup they aren't saved to the new structure.
(cherry picked from commit f7a52bc04e)
Alerting: Fix $value type when single data source is queried (#106080)
(cherry picked from commit faeddf334a)
Co-authored-by: Alexander Akhmetov <me@alx.cx>
Alerting: Ensure field validators return the proper type (#104050)
* Ensure field validators return the proper type
This ensures correct error propagation through services up to
the API layer.
* Move error wrapping up to call site
(cherry picked from commit 820c338414)
Co-authored-by: William Wernert <william.wernert@grafana.com>
What is this feature?
Send resolved notifications not only when an alert state becomes stale (series is missing) and transitions from Alerting to Normal, but also from Error, NoData and Recovering.
Why do we need this feature?
Previously, when an alert state became stale or was deleted, it would transition to Normal but wouldn't trigger resolved notifications to the Alertmanager. This meant we relied on the Alertmanager to send resolved notifications when the alert expires. However, if the Alertmanager state is lost, these resolved notifications would never be sent, leaving users with firing alerts in their notification channels. This PR ensures that any transition from a firing state (Alerting, Error, NoData, Recovering) to Normal triggers a resolved notification.
* Remove POST config for Grafana Alertmanager
* Delete auth + test for removed path
* Alerting: Remove check for `alertingApiServer` toggle in UI (#103805)
* Remove check for alertingApiServer in UI
* Update tests to no longer care about alertingApiServer
* Add contact points handlers now that we use alertingApiServer all the time
* Fix test broken from removing camelCase for UIDs
---------
Co-authored-by: Tom Ratcliffe <tom.ratcliffe@grafana.com>
* Template editor syntax highlighting when preview is json-like
* Add new template editor language examples, snippets, and functions
* Use updated NewTemplate function
* Add new fields to webhook notifier
- CustomPayload
- ExtraHeaders
* Documentation
* Update grafana/alerting to in-progress PR (needs updating after merge)
* Fix integration test
* Remove docs reference to .Extra template context
No longer exists, was part of a previous iteration
* make update-workspace
* Update grafana/alerting to actual merged commit
API Changes:
- Fixes validation in template CRUD API to be closer to how the running
alertmanager will use the template. Should remove some incorrect
validation errors.
- Adds some missing default placeholder labels to receiver testing that
are used during template testing but missing during receiver testing
Template Preview:
- Replaced basic preview with a readonly CodeEditor for better whitespace
and alignment clarity (also adds support for future syntax highlighting
in template previews for upcoming webhook payload templates)
Template Selector (Receiver Form):
- Refactored to use same components as Template editor for preview.
- Fixed preview to work with multi-definition templates
- Fixed copy to correctly copy the template contents instead of
{{ template "<name>" . }}.
Template Editor:
- Fixed detection of when to display functions vs snippets in multi-line
expressions
* Move filtering code to generators for performance reasons
Discarding rules and groups early in the iterable chain limits the number of promises we need to wait for which improves performance significantly
* Add error handling for generators
* Add support for data source filter for GMA rules
* search WIP fix
* Fix datasource filter
* Move filtering back to filtered rules hook, use paged groups for improved performance
* Add queriedDatasources field to grafana managed rules and update filtering logic to rely on it
- Introduced a new field `queriedDatasources` in the AlertingRule struct to track data sources used in rules.
- Updated the Prometheus API to populate `queriedDatasources` when creating alerting rules.
- Modified filtering logic in the ruleFilter function to utilize the new `queriedDatasources` field for improved data source matching.
- Adjusted related tests to reflect changes in rule structure and filtering behavior.
* Add FilterView performance logging
* Improve GMA Prometheus types, rename queried datasources property
* Use custom generator helpers for flattening and filtering rule groups
* Fix lint errors, add missing translations
* Revert test condition
* Refactor api prom changes
* Fix lint errors
* Update backend tests
* Refactor rule list components to improve error handling and data source management
- Enhanced error handling in FilterViewResults by logging errors before returning an empty iterable.
- Simplified conditional rendering in GrafanaRuleLoader for better readability.
- Updated data source handling in PaginatedDataSourceLoader and PaginatedGrafanaLoader to use new individual rule group generator.
- Renamed toPageless function to toIndividualRuleGroups for clarity in prometheusGroupsGenerator.
- Improved filtering logic in useFilteredRulesIterator to utilize a dedicated function for data source type validation.
- Added isRulesDataSourceType utility function for better data source type checks.
- Removed commented-out code in PromRuleDTOBase for cleaner interface definition.
* Fix abort controller on FilterView
* Improve generators filtering
* fix abort controller
* refactor cancelSearch
* make states exclusive
* Load full page in one loadResultPage call
* Update tests, update translations
* Refactor filter status into separate component
* hoist hook
* Use the new function for supported rules source type
---------
Co-authored-by: Gilles De Mey <gilles.de.mey@gmail.com>
This makes it so that it is:
- No longer required to have datasource permissions to delete a rule.
- No longer required to have datasource permissions to update non-query related fields of a rule.
* wip
* Add prom flavor support for data source variables and export/import dashboards (#103321)
* add dashboard and data source var selection
* use match plugin id instead
* use updated matchpluginid
* formatting
* cleanup
* regex anchor
* update error msg
* Alerting: Clean up prometheus-flavored types and functions (#103703)
* clean up types and utility functions for dealing with
prometheus-flavored data sources
* Refactor alerting datasource types to use constants as source of truth
* Alerting: Clean up prometheus-flavored types and functions on the bac… (#103716)
Alerting: Clean up prometheus-flavored types and functions on the backend
* add matchPluginId tests
* Update matchPluginId func to bidirectional (#103746)
* update matchpluginid func to bidirectional
* lint
* formatting
* use actual isSupportedExternalRulesSourceType in test
* add tests in datasource_srv
* betterer
* remove type assertion
* remove unnecessary case
* use satisifies to not have to convert tuple to an array of string
* add prometheus_flavor test
---------
Co-authored-by: Andrew Hackmann <5140848+bossinc@users.noreply.github.com>
Co-authored-by: Gilles De Mey <gilles.de.mey@gmail.com>
Co-authored-by: Alexander Akhmetov <me@alx.cx>
What is this feature?
This PR fixes a state transition issue where alerts transitioning from the Recovering state back to the Alerting state incorrectly entered the Pending state first if the rule had a For duration configured.
Why do we need this feature?
When an alert goes from Alerting to Recovering (when using the Keep firing for) and then back to Alerting, the existing logic would incorrectly put the alert into Pending state while it should be alerting and still sending notifications to the Alertmanager.
* placeholder commit
* Complete function in api_convert_prometheus.go
* MVP before extensive testing
* Cleanup
* Updated tests
* cleanup
* Fix random logs and lint
* Remove comment
* Fix errors after rebase
* Update test
* Update swagger
* swagger
* Refactor to accept groups in body
* Fix auth tests and some cleanup
* Some cleanup before refactoring
* Remove unnecessary fields
* Also refactor RouteConvertPrometheusPostRuleGroup
* Remove unused code
* Rebase + cleanup
* Update authorization_test
* address comments
* Regen swagger files
* Remove namespace and group filters
* Final comments
* remove feature flag alertingAlertmanagerExtraDedupStage
* use most recent version of fork of alertmanager
---------
Co-authored-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
What is this feature?
A follow-up for #101184, adds AlertRule.MissingSeriesEvalsToResolve to the APIs.
missing_series_evals_to_resolve must be specified too and it must be > 0.
POST /api/ruler/grafana/api/v1/rules/{folderUID} works in the following way:
If missing_series_evals_to_resolve is not sent or null, the rule keeps its existing value
If missing_series_evals_to_resolve > 0: updates to that value
If missing_series_evals_to_resolve = 0: resets to default (nil).
AlertRule.MissingSeriesEvalsToResolve can't be 0, so I used it to reset
In the Provisioning API, the value is just set if present and > 0. Otherwise it's reset:
PUT to /api/v1/provisioning/alert-rules/{UID}:
If missing_series_evals_to_resolve is nil, it's reset to the default value
If missing_series_evals_to_resolve > 0, it's updated
What is this feature?
Prometheus conversion: ensures that AlertRule.Data queries always have default parameters set (intervalMs, maxDataPoints). Without this, updates of the same rule can cause version increments.
Why do we need this feature?
Currently, when converting Prometheus rules to Grafana alerts, some default parameters are not explicitly set in the query model. This creates a problem during rule updates:
When a user updates a rule that hasn't changed, we still detect differences in the AlertQuery.Model because the newly converted rules are missing the default fields, such as intervalMs and maxDataPoints. This causes unnecessary version increments of alert rules.
What is this feature?
This PR changes the behavior of the $value and .Value variables in alerting templating to be more compatible with Prometheus templating. When a single datasource is used in the alerting rule, these variables will now return the numeric value from the query instead of the evaluation string.
Why do we need this feature?
It makes Grafana templating more compatible with Prometheus templates. In Prometheus, $value returns the numeric value of the query, but in Grafana it's the evaluation string: [ var='A' labels={instance=instance1} value=81.234 ]. This is because in Grafana multiple datasources can be used in the alert rule, and it's not always possible to get a single value.
This change makes Grafana's behavior consistent with Prometheus when a single datasource is used, and in case when multiple datasources are used in the query, it keeps the old behaviour.
Both $value and .Value are not recommended to use (documentation), and it's better to use .Values instead.
* Add dashboard cleanup job
Change log message
Adjust logic to account for new head RV logic
Don't update lastResourceVersion due to pagination
Save improvements
* Address review feedback
* Update docs.
* Remove docs
* Rename config
---------
Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
Adds support for rule group-level query_offset in Prometheus to Grafana rule conversion. It allows specifying a time offset for rule evaluation, which gets applied and saved during the conversion.
Adds a new configuration option to specify a time offset for rule evaluation, which gets applied and saved during the Prometheus -> Grafana conversion. For example:
[unified_alerting.prometheus_conversion]
rule_query_offset = 1m
Changing this option affects only the rules imported after the change. If query_offset is set at the group level, it takes precedence over this setting.
Default is set to 1m.
* Alerting: Fix loss of TimeInterval location on remote AM apply
deepcopy.Copy does not correctly copy PostableUserConfig because it ignores
unexported fields. As a result, TimeInterval locations default to UTC instead
of retaining their original values.
* make update-workspace
Moustafa Baiou
Alexander Akhmetov
github-actions[bot]
Matheus Macabu
grafana-delivery-bot[bot]
Stephanie Hingtgen
William Wernert
Matthew Jacobson
Konrad Lalik
Kevin Yu
Mariell Hoversholm
Steve Simpson
Will Assis
Will Browne
Yuri Tseretyan
maicon
Fayzal Ghantiwala
alerting-team[bot]
Peter Štibraný
Marco de Abreu