Recording rule fields were not being copied correctly when duplicating an alert rule. This manifests as missing `TargetDataSourceUID` fields from the `Record` part of the rule when rules in a group are re-ordered.
Added some additional tests to ensure we cover the generation of recording rules in tests and fixed the copying logic to ensure all fields are copied correctly.
(cherry picked from commit c73b3ccf6e)
Alerting: Ensure field validators return the proper type (#104050)
* Ensure field validators return the proper type
This ensures correct error propagation through services up to
the API layer.
* Move error wrapping up to call site
(cherry picked from commit 820c338414)
Co-authored-by: William Wernert <william.wernert@grafana.com>
What is this feature?
A follow-up for #101184, adds AlertRule.MissingSeriesEvalsToResolve to the APIs.
missing_series_evals_to_resolve must be specified too and it must be > 0.
POST /api/ruler/grafana/api/v1/rules/{folderUID} works in the following way:
If missing_series_evals_to_resolve is not sent or null, the rule keeps its existing value
If missing_series_evals_to_resolve > 0: updates to that value
If missing_series_evals_to_resolve = 0: resets to default (nil).
AlertRule.MissingSeriesEvalsToResolve can't be 0, so I used it to reset
In the Provisioning API, the value is just set if present and > 0. Otherwise it's reset:
PUT to /api/v1/provisioning/alert-rules/{UID}:
If missing_series_evals_to_resolve is nil, it's reset to the default value
If missing_series_evals_to_resolve > 0, it's updated
What is this feature?
Prometheus conversion: ensures that AlertRule.Data queries always have default parameters set (intervalMs, maxDataPoints). Without this, updates of the same rule can cause version increments.
Why do we need this feature?
Currently, when converting Prometheus rules to Grafana alerts, some default parameters are not explicitly set in the query model. This creates a problem during rule updates:
When a user updates a rule that hasn't changed, we still detect differences in the AlertQuery.Model because the newly converted rules are missing the default fields, such as intervalMs and maxDataPoints. This causes unnecessary version increments of alert rules.
What is this feature?
This PR introduces a new alert rule configuration option, keep_firing_for (Prometheus documentation).
keep_firing_for prevents alerts from resolving immediately after the alert condition returns to normal. Instead, they transition into a "Recovering" state and are not considered resolved by the Alertmanager. Once the recovery period ends (or after the next evaluation if it is bigger than keep_firing_for), the alert transitions to "Normal" if it doesn't start alerting again:
Before
+----------+ +----------+
| Alerting |---->| Normal |
+----------+ +----------+
-----
After
+----------+ +------------+ +----------+
| Alerting |----->| Recovering |---->| Normal |
+----------+ +------------+ +----------+
Why do we need this feature?
This feature prevents flapping alerts by adding a recovery period. This helps avoid false resolutions caused by brief alert
Extend the recording rule definition to include the target data source, allowing
configuration of where the output of the recording rule is written to. Also
extends the relevant interfaces in preparation for the next set of changes.
* add column guid to alert rule table and rule_guid to rule version table
+ populate the new field with UUID
* update storage and domain models
* patch GUID
* ignore GUID in fingerprint tests
Initially, Metadata had only the EditorSettings, and HasMetadata was used to understand if the incoming update request had metadata in the body because it could be omitted if it was empty. For example, when the rule is updated via the provisioning API or has only false values. If it was in the request, we used that; if not, we used the metadata from the existing rule from the database. If the rule was updated via the AlertRuleService, we didn't change Metadata at all if the rule already existed.
But now, Metadata also has the Prometheus rule definition, and we always need to update it with the new version of the AlertRuleService when the rule exists in the DB and has the same UID. HasMetadata is renamed to HasEditorSettings to keep the old behaviour only for EditorSettings.
Now, the provisioning API and the conversion API will overwrite everything except EditorSettings with the new data.
What is this feature?
Adds an API endpoint to create alert rules with mimirtool:
- POST /convert/prometheus/config/v1/rules/{NamespaceTitle} - Accepts a single rule group in a Prometheus YAML format and creates or updates a Grafana rule group from it.
The endpoint uses the conversion package from #100224.
Key parts
The API works similarly to the provisioning API. If the rule does not exist, it will be created, otherwise updated. Any rules not present in the new group will be deleted, ensuring the group is fully synchronized with the provided configuration.
Since the API works with namespace titles (folders), the handler automatically creates a folder in the root based on the provided title if it does not exist. It also requires a special header, X-Grafana-Alerting-Datasource-UID. This header specifies which datasource to use for the new rules.
If the rule group's evaluation interval is not specified, it uses the DefaultRuleEvaluationInterval from settings.
* Send new annotation containing image url
* Use new image TokenProvider with TokenStore
New abstraction GetImage no longer needs to support parsing both token and
url from annotations, as remote AM will use the new URLProvider. Instead, we
use the new generic TokenProvider and give it a TokenStore backed by the
grafana database.
That means we revert back to always using token simplifying code and security
considerations.
* Upgrade grafana/alerting to merged commit SHA
* Alerting: Fix Alertmanager configuration updates
Alertmanager configuration updates would behave inconsistently when performing no-op updates with `mysql` as the store.
In particular this bug manifested as a failure to reload the provisioned alertmanager configuration components with no changes to the configuration itself. This would result in a 500 error with mysql store only.
The core issue is that we were relying on the number of rows affected by the update query to determine if the configuration was found in the db or not.
While this behavior works for certain sql dialects, mysql does not return the number of rows matched by the update query but rather the number of rows actually updated.
Also discovered and fixed the mismatched `xorm` tag for the `CreatedAt` field to match the actual column name in the db.
References: https://dev.mysql.com/doc/refman/8.4/en/update.html
* introduce new fields created_by in rule tables
* update domain model and compat layer to support UpdatedBy
* add alert rule generator mutators for UpdatedBy
* ignore UpdatedBy in diff and hash calculation
* Add user context to alert rule insert/update operations
Updated InsertAlertRules and UpdateAlertRules methods to accept a user context parameter. This change ensures auditability and better tracking of user actions when creating or updating alert rules. Adjusted all relevant calls and interfaces to pass the user context accordingly.
* set UpdatedBy in PreSave because this is where Updated is set
* Use nil userID for system-initiated updates
This ensures differentiation between system and user-initiated changes for better traceability and clarity in update origins.
---------
Signed-off-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
* Add health fields to rules and an aggregator method to the scheduler
* Move health, last error, and last eval time in together to minimize state processing
* Wire up a readonly scheduler to prom api
* Extract to exported function
* Use health in api_prometheus and fix up tests
* Rename health struct to status
* Fix tests one more time
* Several new tests
* Handle inactive rules
* Push state mapping into state manager
* rename to StatusReader
* Rectify cyclo complexity rebase
* Convert existing package local status implementation to models one
* fix tests
* undo RuleDefs rename
* Include access control metadata in k8s receiver List & Get
* Add tests for receiver access
* Simplify receiver access provisioning extension
- prevents edge case infinite recursion
- removes read requirement from create
* introduce storage model for alert rule tables
* remove AlertRuleVersion from models because it's not used anywhere other than in storage
* update historian xorm store to use alerting store to fetch rules
* fix folder tests
---------
Co-authored-by: Matthew Jacobson <matthew.jacobson@grafana.com>
Back-end:
* update alerting module
* update GetSecretKeysForContactPointType to extract secret fields from nested options
* Update RemoveSecretsForContactPoint to support complex settings
* update PostableGrafanaReceiverToEmbeddedContactPoint to support nested secrets
* update Integration to support nested settings in models.Integration
* make sigv4 fields optional
Front-end:
* add UI support for encrypted subform fields
* allow emptying nested secure fields
* Omit non touched secure fields in POST payload when saving a contact point
* Use SecretInput from grafana-ui instead of the new EncryptedInput
* use produce from immer
* rename mapClone
* rename sliceClone
* Don't use produce from immer as we need to delete the fileds afterwards
---------
Co-authored-by: Gilles De Mey <gilles.de.mey@gmail.com>
Co-authored-by: Sonia Aguilar <soniaaguilarpeiron@gmail.com>
Co-authored-by: Matt Jacobson <matthew.jacobson@grafana.com>
* Replace global authz abstraction with one compatible with uid scope
* Replace GettableApiReceiver with models.Receiver in receiver_svc
* GrafanaIntegrationConfig -> models.Integration
* Implement Create/Update methods
* Add optimistic concurrency to receiver API
* Add scope to ReceiversRead & ReceiversReadSecrets
migrates existing permissions to include implicit global scope
* Add receiver create, update, delete actions
* Check if receiver is used by rules before delete
* On receiver name change update in routes and notification settings
* Improve errors
* Linting
* Include read permissions are requirements for create/update/delete
* Alias ngalert/models to ngmodels to differentiate from v0alpha1 model
* Ensure integration UIDs are valid, unique, and generated if empty
* Validate integration settings on create/update
* Leverage UidToName to GetReceiver instead of GetReceivers
* Remove some unnecessary uses of simplejson
* alerting.notifications.receiver -> alerting.notifications.receivers
* validator -> provenanceValidator
* Only validate the modified receiver
stops existing invalid receivers from preventing modification of a valid
receiver.
* Improve error in Integration.Encrypt
* Remove scope from alert.notifications.receivers:create
* Add todos for receiver renaming
* Use receiverAC precondition checks in k8s api
* Linting
* Optional optimistic concurrency for delete
* make update-workspace
* More specific auth checks in k8s authorize.go
* Add debug log when delete optimistic concurrency is skipped
* Improve error message on authorizer.DecisionDeny
* Keep error for non-forbidden errutil errors
* add RenameTimeIntervalInNotificationSettings to storage
* update dependencies when the time interval is renamed
---------
Co-authored-by: William Wernert <william.wernert@grafana.com>
* Alerting: Add rule_group label to grafana_alerting_rule_group_rules metric (#62361)
* Alerting: Delete rule group metrics when the rule group is deleted
This commit addresses the issue where the GroupRules metric (a GaugeVec)
keeps its value and is not deleted when an alert rule is removed from the rule registry.
Previously, when an alert rule with orgID=1 was active, the metric was:
grafana_alerting_rule_group_rules{org="1",state="active"} 1
However, after deleting this rule, subsequent calls to updateRulesMetrics
did not update the gauge value, causing the metric to incorrectly remain at 1.
The fix ensures that when updateRulesMetrics is called it
also deletes the group rule metrics with the corresponding label values if needed.
Moustafa Baiou
grafana-delivery-bot[bot]
Mariell Hoversholm
Yuri Tseretyan
maicon
Alexander Akhmetov
Steve Simpson
Matthew Jacobson
Garret Wyman
Santiago
Nihal
William Wernert
Alexander Weaver