* Annotations: Split cleanup into separate queries and deletes to avoid deadlocks on MySQL (#80329)
* Split subquery when cleaning annotations
* update comment
* Raise batch size, now that we pay attention to it
* Iterate in batches
* Separate cancellable batch implementation to allow for multi-statement callbacks, add overload for single-statement use
* Use split-out utility in outer batching loop so it respects context cancellation
* guard against empty queries
* Use SQL parameters
* Use same approach for tags
* drop unused function
* Work around parameter limit on sqlite for large batches
* Bulk insert test data in DB
* Refactor test to customise test data creation
* Add test for catching SQLITE_MAX_VARIABLE_NUMBER limit
* Turn annotation cleanup test to integration tests
* lint
---------
Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
(cherry picked from commit 81c45bfe44)
* Fix interval, logs per backport
* empty commit to kick actions
Release: Deprecate latest.json and replace with api call to grafana.com (#80537)
* remove latest.json and replace with api call to grafana.com
* remove latest.json
* Revert "remove latest.json"
This reverts commit bcff43d898.
* Revert "remove latest.json and replace with api call to grafana.com"
This reverts commit 02b867d84e.
* add deprecation message to latest.json
(cherry picked from commit 127decee1e)
Auth: id response header (#77871)
* Add config options for identity id response header
* Add feature to add identity id response header to all responses
* Use util.SplitString
(cherry picked from commit 21f94c5b78)
Alerting: Attempt to retry retryable errors (#79161)
* Alerting: Attempt to retry retryable errors
Retrying has been broken for a good while now (at least since version 9.4) - this change attempts to re-introduce them in their simplest and safest form possible.
I first introduced #79095 to make sure we don't disrupt or put additional load on our customer's data sources with this change in a patch release. Paired with this change, retries can now work as expected.
There's two small differences between how retries work now and how they used to work in legacy alerting.
Retries only occur for valid alert definitions - if we suspect that that error comes from a malformed alert definition we skip retrying.
We have added a constant backoff of 1s in between retries.
---------
Signed-off-by: gotjosh <josue.abreu@gmail.com>
(cherry picked from commit c631261681)
Alerting: Only warm alert state cache if execute_alerts=true. (#78895)
* Alerting: Only warm alert state cache if execute_alerts=true.
If the Grafana instance is not executing alerts, then Warm()-ing the state
manager is wasteful and could lead to misleading rule status queries, as the
status returned will be always based on the state loaded from the database at
startup, and not the most recent evaluation state.
* Move Warm() down to shared conditional.
(cherry picked from commit 520c927931)
Co-authored-by: Steve Simpson <steve.simpson@grafana.com>
* [LDAP] Disable removed users on login (#74016)
* [LDAP] Disable removed users on login
* Fix tests
* Add test for user disabling
* Add tests for disabling user behind auth proxy
* Linting.
* Rename setup func
* Account for reviews comments
Co-authored-by: Kalle Persson <kalle.persson@grafana.com>
---------
Co-authored-by: Kalle Persson <kalle.persson@grafana.com>
(cherry picked from commit f900098cc9)
* manual backport of #74016
* LDAP: Fix active sync with large quantities of users (#73834)
* Fix middleware test
---------
Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>
Co-authored-by: Gabriel MABILLE <gabriel.mabille@grafana.com>
* SSE: DSNode to update result with names to make each value identifiable by labels (only Graphite and TestData) (#71246)
* introduce a function checkIfSeriesNeedToBeFixed to scan all value fields in the response and provide a function that updates Series so they can be uniquely identifiable. Only Graphite and TestData are checked.
* update `DSNode.Execute` to run this function and provide it to WideToMany
* update WideToMany to run the fix function if it is not nil
* Alerting: Fix state manager to not keep datasource_uid and ref_id labels in state after Error (#72216)
(cherry picked from commit 78fc3bcdf4)
* use old QueryError
---------
Co-authored-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
Chore: Remove topnav feature flag (#72337)
* Remove topnav feature flag
* Allow deprecated flags to be enabled by default
* change topnav feature flag to deprecated instead
* fix lint
(cherry picked from commit 488eac0e49)
* Alerting: Fix contact point testing with secure settings (#72235)
Fixes double encryption of secure settings during contact point testing and removes code duplication
that helped cause the drift between alertmanager and test endpoint. Also adds integration tests to cover
the regression.
Note: provisioningStore is created to remove cycle and the unnecessary dependency.
(cherry picked from commit d31d175109)
* Alerting: Improve performance of matching captures (#71828)
This commit updates eval.go to improve the performance of matching
captures in the general case. In some cases we have reduced the
runtime of the function from 10s of minutes to a couple 100ms.
In the case where no capture matches the exact labels, we revert to
the current subset/superset match, but with a reduced search space
due to grouping captures.
(cherry picked from commit 8dd3eb856d)
* Add label fingerprints from grafana-plugin-sdk-go
* Remove unsafe.StringData as we use Go 1.19
* Fix lint
---------
Co-authored-by: George Robinson <george.robinson@grafana.com>
Alerting: Add tests for matching captures (#71928)
This commit adds tests for matching captures, which we do not have
at present.
(cherry picked from commit f1af0502db)
Co-authored-by: George Robinson <george.robinson@grafana.com>
Alerting: Sort NumberCaptureValues in EvaluationString (#71927)
This commit changes extractEvalString to sort NumberCaptureValues
in ascending order of Var before building the output string. This
means that users will see EvaluationString in a consistent order,
but also make it possible to assert its output in tests.
(cherry picked from commit 89dcaaf049)
Co-authored-by: George Robinson <george.robinson@grafana.com>
Alerting: No longer silence paused alerts during legacy migration (#71596)
* Alerting: No longer silence paused alerts during legacy migration
Now that we migrate paused legacy alerts to paused UA alert rules, we no longer need to silence them.
(cherry picked from commit 8c6cdf51fc)
Co-authored-by: Matthew Jacobson <matthew.jacobson@grafana.com>
Alerting: Fix Alertmanager change detection for receivers with secure settings (#71307)
* Alerting: Make ApplyAlertmanagerConfiguration only decrypt/encrypt new/changed secure settings
Previously, ApplyAlertmanagerConfiguration would decrypt and re-encrypt all secure settings. However, this caused re-encrypted secure settings to be included in the raw configuration when applied to the embedded alertmanager, resulting in changes to the hash. Consequently, even if no actual modifications were made, saving any alertmanager configuration triggered an apply/restart and created a new historical entry in the database.
To address the issue, this modifies ApplyAlertmanagerConfiguration, which is called by POST `api/alertmanager/grafana/config/api/v1/alerts`, to decrypt and re-encrypt only new and updated secure settings. Unchanged secure settings are loaded directly from the database without alteration.
We determine whether secure settings have changed based on the following (already in-use) assumption: Only new or updated secure settings are provided via the POST `api/alertmanager/grafana/config/api/v1/alerts` request, while existing unchanged settings are omitted.
* Ensure saving a grafana-managed contact point will only send new/changed secure settings
Previously, when saving a grafana-managed contact point, empty string values were transmitted for all unset secure settings. This led to potential backend issues, as it assumed that only newly added or updated secure settings would be provided.
To address this, we now exclude empty ('', null, undefined) secure settings, unless there was a pre-existing entry in secureFields for that specific setting. In essence, this means we only transmit an empty secure setting if a previously configured value was cleared.
* Fix linting
* refactor omitEmptyUnlessExisting
* fixup
---------
Co-authored-by: Gilles De Mey <gilles.de.mey@gmail.com>
(cherry picked from commit e3787de470)
Co-authored-by: Matthew Jacobson <matthew.jacobson@grafana.com>
Fix: Change getExistingDashboardByTitleAndFolder to get dashboard by title, not slug (#70723)
* Fix: Change getExistingDashboardByTitleAndFolder to get dashboard by title, not slug
* test: add tests for get dashboard with existing name, get dashboard with non existing name, get dashboard with existing name in a folder
* Update pkg/services/dashboards/database/database_test.go
Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
* require specific error for Should be able to get dashboard with existing name
* Update pkg/services/dashboards/database/database_test.go
Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
* implement sofia suggestions to check for specific err, remove logs
* give test more specific name
* implement daniel suggestion of formatting documentation comment in safe way
* fix test title to refer to root directory not specific folder
---------
Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
(cherry picked from commit 67cdae4b7d)
Co-authored-by: Kat Yang <69819079+yangkb09@users.noreply.github.com>
* Alerting: Support concurrent queries for saving alert instances (#70525)
This commit adds support for concurrent queries when saving alert
instances to the database. This is an experimental feature in
response to some customers experiencing delays between rule evaluation
and sending alerts to Alertmanager, resulting in flapping. It is
disabled by default.
(cherry picked from commit 7edbe72483)
* remove changes in api_testing.go
---------
Co-authored-by: George Robinson <george.robinson@grafana.com>
Alerting: Convert 'Both' type Prometheus queries to 'Range' in migration (#70781)
* Alerting: Convert 'Both' type Prometheus queries to 'Range' in migration
(cherry picked from commit 00d5f7fed7)
Co-authored-by: Matthew Jacobson <matthew.jacobson@grafana.com>
Alerting: Add limit query parameter to Loki-based ASH api, drop default limit from 5000 to 1000, extend visible time range for new ASH UI (#70769)
* Add limit query parameter
* Drop copy paste comment
* Extend history query limit to 30 days and 250 entries
* Fix history log entries ordering
* Update no history message, add empty history test
---------
Co-authored-by: Konrad Lalik <konrad.lalik@grafana.com>
(cherry picked from commit f94fb765b5)
Co-authored-by: Alexander Weaver <weaver.alex.d@gmail.com>
Alerting: Move rule UID from Loki stream labels into log lines (#70637)
Move rule uid into log line to reduce cardinality
(cherry picked from commit 4aa477f48f)
Co-authored-by: William Wernert <william.wernert@gmail.com>
* Alerting: Fix TestIntegration_GetAlertRulesForScheduling to make sure rules are created in different org (#69088)
make sure rules are created in different org
(cherry picked from commit b57ef1f2c7)
* Alerting: Fix flaky TestIntegrationUpdateAlertRules (#69106)
Prevents duplicate alert rule ids and 0 value for BaseInterval and IntervalSeconds
(cherry picked from commit 97ae6ae6ef)
* Alerting: Fix unique violation when updating rule group with title chains/cycles (#67868)
* Alerting: Fix unique violation when updating rule group with title chains/cycles
The uniqueness constraint for titles within an org+folder is enforced on every update within a transaction instead of on commit (deferred constraint). This means that there could be a set of updates that will throw a unique constraint violation in an intermediate step even though the final state is valid. For example, a chain of updates RuleA -> RuleB -> RuleC could fail if not executed in the correct order, or a swap of titles RuleA <-> RuleB cannot be executed in any order without violating the constraint.
The exact solution to this is complex and requires determining directed paths and cycles in the update graph, adding in temporary updates to break cycles, and then executing the updates in reverse topological order (see first commit in PR if curious).
This is not implemented here.
Instead, we choose a simpler solution that works in all cases but might perform more updates than necessary. This simpler solution makes a determination of whether an intermediate collision could occur and if so, adds a temporary title on all updated rules to break any cycles and remove the need for specific ordering.
In addition, we make sure diffs are executed in the following order: DELETES, UPDATES, INSERTS.
(cherry picked from commit 0c688190f7)
* Linting
---------
Co-authored-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
Alerting: Add debug logs for EndsAt timestamp (#70336)
This commit adds debug logs for previous_ends_at and next_ends_at
to state.go to help us debug issues where alerts are resolved in
Alertmanager due to expiration. This change is in response to a
support escalation where this information was needed but unavailable.
(cherry picked from commit 815e98ed95)
Co-authored-by: George Robinson <george.robinson@grafana.com>
* Alerting: Add image URI annotation only when there's an image (#69825)
* Alerting: Add image URI annotation only when there's an image
* fix function name (changed on main branch)
(cherry picked from commit ff3e028a85)
* Update compat_test.go
---------
Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
Service accounts: API key migration refactoring to parse as json object of the results (#69771)
refactoring to parse as json object of the results
(cherry picked from commit 5dceb5dff3)
Co-authored-by: Eric Leijonmarck <eric.leijonmarck@gmail.com>
Alerting: Change text on cloud AM email addresses for contact points (#68143)
(cherry picked from commit e1ff434917)
Co-authored-by: Virginia Cepeda <virginia.cepeda@grafana.com>
Service accounts: UI migration results (#68789)
* ui migration WIP
* merge
* migration tests for api
* revert chagnes to align with main
* revert chagnes to align with main
* revert chagnes to align with main
* remove unused code and comments
* revert gen files
* retry logic inplace
* fix a any
* fixed types
* migraiton results now show only result if no failures
* review comments
* wording to make it more actionable
* add migraiton summary text onyl for failed apikeys
* fixed wording and added a close button to the modal
* made the button close the modal
* moved state into component
* fix based on review, naming and removed unused code
* service account migration state optional
* making migration result undefined
* showing total and migrated numbers for a successful migration
* fix payload const to take the payload
(cherry picked from commit 081f59feba)
Co-authored-by: Eric Leijonmarck <eric.leijonmarck@gmail.com>
Prevent crash while executing concurrent mixed queries (#874)
limit parallel query execution to 1 at a time
(cherry picked from commit 96579a60e19e2a9f2d6bdaeba64e0e702211eb73)
Co-authored-by: Michael Mandrus <41969079+mmandrus@users.noreply.github.com>
Require alert.notifications:write permissions to test receivers and templates (#865)
(cherry picked from commit 3c21ab70075256d4ba8e4fbfdcb15f5a394161fa)
Co-authored-by: George Robinson <george.robinson@grafana.com>
Alerting: Fix provisioned templates being ignored by alertmanager (#69485)
* Alerting: Fix provisioned templates being ignored by alertmanager
Template provisioning sets the template in cfg.TemplateFiles while a recent change
made it so that alertmanager reads cfg.AlertmanagerConfig.Templates instead.
This change fixes the issue on both ends, by having provisioning set boths fields and
reverts the change on the alertmanager side so that it uses cfg.TemplateFiles.
(cherry picked from commit c16f1f5e99)
SQLStore: Align SQLite IsUniqueConstraintViolation() with other backend implementations (#69224)
* Add integration test for primary key and unique constrain violation
* Align SQLite IsUniqueConstraintViolation implementation with other backends
(cherry picked from commit 74e87ccbbd)
Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
Alerting: Fix provenance guard checks for Alertmanager configuration to not cause panic when compared nested objects (#69009)
* fix current settings parsed as new
* replace map comparison with cmp.Diff and log the diff
(cherry picked from commit e00260465b)
Co-authored-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
Chore: Ignore unique constrain failure when creating the main organization (#68644)
Chore: Ignore unique constrain failure when creatin the main organization
(cherry picked from commit 633b7875e7)
Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
Alexander Weaver
Ashley Harrison
Karl Persson
gotjosh
grafana-delivery-bot[bot]
Yuri Tseretyan
Josh Hunt
Matthew Jacobson
Santiago
Ieva
Grot (@grafanabot)
Jean-Philippe Quéméner
Alexander Zobnin