Part 2 of adding version messages to the `alert_rule_version` able. This allows
setting the message via a header when using the Prometheus conversion API,
which can be useful for e.g. linking changes back to source control.
* fix: collate rule_group column as utf8mb4_bin
This ensures this column is sorted identically by the database and
golang, which should eliminate issues related to a mismatch in sort
order.
* chore: make test stricter
* fix: also add binary collation for postgres
* Deprecating features.IsEnabled
* add one more nolint
* add one more nolint
* Give better hints to devs in the deprecation message of IsEnabledGlobally
* adding more doc strings
* fix linter after rebase
* Extend deprecation message
This adds a `message` column to the `alert_rule_version` table. This follows the
pattern established for dashboards as closely as possible. A new type is
introduced internally for passing the new `message` field around in a type-safe
manner, but doing the same for the API types becomes very messy. In that case, a
new field is added with omitempty.
Note this PR is only:
- The column addition
- The "read" path; API for listing versions
Subsequent PRs will add code to actually set the message when updating rules.
What is this feature?
This PR implements compressed periodic save for alert state storage, providing a more efficient alternative to regular periodic saves by grouping alert instances by rule UID and storing them using protobuf and snappy compression. When enabled via the state_compressed_periodic_save_enabled configuration option, the system groups alert instances by their alert rule, compresses each group using protobuf serialization and snappy compression, and processes all rules within a single database transaction at specified intervals instead of syncing after every alert evaluation cycle.
Why do we need this feature?
During discussions in PR #111357, we identified the need for a compressed approach to periodic alert state storage that could further reduce database load beyond the jitter mechanism. While the jitter feature distributes database operations over time, this compressed periodic save approach reduces the frequency of database operations by batching alert state updates at explicitly declared intervals rather than syncing after every alert evaluation cycle.
This approach provides several key benefits:
- Reduced Database Frequency: Instead of frequent sync operations tied to alert evaluation cycles, updates occur only at configured intervals
- Storage Efficiency: Rule-based grouping with protobuf and snappy compression significantly reduces storage requirements
The compressed periodic save complements the existing jitter mechanism by providing an alternative strategy focused on reducing overall database interaction frequency while maintaining data integrity through compression and batching.
Who is this feature for?
- Platform/Infrastructure teams managing large-scale Grafana deployments with high alert cardinality
- Organizations looking to optimize storage costs and database performance for alerting workloads
- Production environments with 1000+ alert rules where database write frequency is a concern
The query which fetches alert rules in a paginated manner ordered by `rule_group` can result in strange and inconsistent ordering when the database uses a case-insensitive collation for the `rule_group` column. This can lead to scenarios where rules from different groups are interleaved in the results, making pagination unreliable and the returned number of rule_groups incorrect.
Related to #88990
What is this feature?
This PR implements a jitter mechanism for periodic alert state storage to distribute database load over time instead of processing all alert instances simultaneously. When enabled via the state_periodic_save_jitter_enabled configuration option, the system spreads batch write operations across 85% of the save interval window, preventing database load spikes in high-cardinality alerting environments.
Why do we need this feature?
In production environments with high alert cardinality, the current periodic batch storage can cause database performance issues by processing all alert instances simultaneously at fixed intervals. Even when using periodic batch storage to improve performance, concentrating all database operations at a single point in time can overwhelm database resources, especially in resource-constrained environments.
Rather than performing all INSERT operations at once during the periodic save, distributing these operations across the time window until the next save cycle can maintain more stable service operation within limited database resources. This approach prevents resource saturation by spreading the database load over the available time interval, allowing the system to operate more gracefully within existing resource constraints.
For example, with 200,000 alert instances using a 5-minute interval and 4,000 batch size, instead of executing 50 batch operations simultaneously, the jitter mechanism distributes these operations across approximately 4.25 minutes (85% of 5 minutes), with each batch executed roughly every 5.2 seconds.
This PR provides system-level protection against such load spikes by distributing operations across time, reducing peak resource usage while maintaining the benefits of periodic batch storage. The jitter mechanism is particularly valuable in resource-constrained environments where maintaining consistent database performance is more critical than precise timing of state updates.
This lets the prometheus api respect NoGroup query logic and treat non-grouped rules consistently.
Co-authored-by: William Wernert <william.wernert@grafana.com>
Rules created in the new api makes the rule have no group in the database, but the rule is returned in the old group api with a sentinel group name formatted with the rule uid for compatiblity with the old api.
This makes the UI continue to work with the rules without a group, and the ruler will continue to work with the rules without a group.
Rules are not allowed to be created in the provisioning api with a NoGroup sentinel mask, but NoGroup rules can be manipulated through both the new and old apis.
Co-authored-by: William Wernert <william.wernert@grafana.com>
* Replace remaining calls to testing.Short where possible.
* Update style guide.
* Revert change in TestAlertmanager_ExtraDedupStage, as it doesn't work.
* Make TestAlertRulePostExport into integration test.
* Alerting: Add Extended List Query for Alert Rules w/pagination
This adds an extended query which allows filtering by the kind of rule (Recording or Alerting) and supports pagination.
Pagination tokens will allow us to list all rules paginated, regardless of the rule group.
---------
Co-authored-by: William Wernert <william.wernert@grafana.com>
This reintroduces store level pagination, without using it in the prometheus API yet.
Related to #108633
Co-authored-by: William Wernert <william.wernert@grafana.com>
* Alerting: Move some tests to full integration tests
* add back two error handling tests
* imports
---------
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
What is this feature?
This PR further improves concurrent updates to the provenance table (a follow-up for #101688). The fix explicitly checks for a provenance record using SELECT ... FOR UPDATE for the databases that support it before performing either an update or upsert operation. This preemptive locking reduces the possibility of deadlocks in MySQL.
This works only with the alertingProvenanceLockWrites feature flag enabled.
Why do we need this feature?
The current implementation (directly performing upserts without prior locking) may still encounter deadlocks because of the gap and insert-intention locks in some configurations, for example, with repeatable-read transaction isolation level in MySQL+InnoDB.
---------
Co-authored-by: William Wernert <william.wernert@grafana.com>
* automatically rename integration tests to follow the common convention
* name tests differently
* alter column type to bigint
* update another column to bigint
* add another alter
* fix subquery for mysql
* Dependencies: Bump github.com/openfga/openfga from v1.8.6 to v1.8.12
* Linter: Replace x/exp/rand with math/rand/v2
* NGAlert: Fix test after linter fixes
* add active_time_intervals to route model
* update k8s compat layer
* update notification policies service to validate active time intervals
* update integration tests
* update openapi
* add active time interval to model
* update route generator to include active time interval
* Update storage list and rename methods to handle active intervals
* update api model
* update provisioning and export models
* update ui to allow active timing config
* update i18n
* fix snapshots for ui tests
* run prettier
* Alerting: Active time intervals UI naming (#104402)
* update naming in UI
* update naming in the edit page title
* update translations
* update alerting module
---------
Signed-off-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
Co-authored-by: Sonia Aguilar <33540275+soniaAguilarPeiron@users.noreply.github.com>
Co-authored-by: Sonia Aguilar <soniaaguilarpeiron@gmail.com>
* replace xorm.io/xorm imports
* replace xorm from other go.mod files
* clean up workspace
* nolint does not make sense anymore as it is not a module
* try if nolint directive helps
* use nolint:all for xorm
* add more nolints
* try to skip xorm in linter config
* exclude xorm differently
* retrigger ci
* Fix TestIntegrationUpdatingProvisionionedDashboards* tests when running on Spanner by fixing case-insensitive search for dashboard title.
* Fix use of case-insensitive LIKE when running on Spanner.
* Fix use of LikeStr in anonstore.
* Fix use of LikeStr in ngalert/store and org/orgimpl.
* Fix use of LikeStr in queryhistory search.
* Fix use of LikeStr in serviceaccounts.
* Fix use of LikeStr in serviceaccounts.
* Fix use of LikeStr in services/team.
* Remove LikeStr method.
This makes it so that it is:
- No longer required to have datasource permissions to delete a rule.
- No longer required to have datasource permissions to update non-query related fields of a rule.
Alexander Akhmetov
Santiago
Steve Simpson
William Wernert
Denis Vodopianov
Seunghun Shin
Moustafa Baiou
Peter Štibraný
Ryan McKinley
Yuri Tseretyan
Fayzal Ghantiwala
Stephanie Hingtgen
Serge ZaitsevandMatheus Macabu
Mariell Hoversholm
maicon