* Annotations: Split cleanup into separate queries and deletes to avoid deadlocks on MySQL (#80329)
* Split subquery when cleaning annotations
* update comment
* Raise batch size, now that we pay attention to it
* Iterate in batches
* Separate cancellable batch implementation to allow for multi-statement callbacks, add overload for single-statement use
* Use split-out utility in outer batching loop so it respects context cancellation
* guard against empty queries
* Use SQL parameters
* Use same approach for tags
* drop unused function
* Work around parameter limit on sqlite for large batches
* Bulk insert test data in DB
* Refactor test to customise test data creation
* Add test for catching SQLITE_MAX_VARIABLE_NUMBER limit
* Turn annotation cleanup test to integration tests
* lint
---------
Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
(cherry picked from commit 81c45bfe44)
* Fix interval, logs per backport
* empty commit to kick actions
Release: Deprecate latest.json and replace with api call to grafana.com (#80537)
* remove latest.json and replace with api call to grafana.com
* remove latest.json
* Revert "remove latest.json"
This reverts commit bcff43d898.
* Revert "remove latest.json and replace with api call to grafana.com"
This reverts commit 02b867d84e.
* add deprecation message to latest.json
(cherry picked from commit 127decee1e)
Auth: id response header (#77871)
* Add config options for identity id response header
* Add feature to add identity id response header to all responses
* Use util.SplitString
(cherry picked from commit 21f94c5b78)
Alerting: Attempt to retry retryable errors (#79161)
* Alerting: Attempt to retry retryable errors
Retrying has been broken for a good while now (at least since version 9.4) - this change attempts to re-introduce them in their simplest and safest form possible.
I first introduced #79095 to make sure we don't disrupt or put additional load on our customer's data sources with this change in a patch release. Paired with this change, retries can now work as expected.
There's two small differences between how retries work now and how they used to work in legacy alerting.
Retries only occur for valid alert definitions - if we suspect that that error comes from a malformed alert definition we skip retrying.
We have added a constant backoff of 1s in between retries.
---------
Signed-off-by: gotjosh <josue.abreu@gmail.com>
(cherry picked from commit c631261681)
Unified Alerting: Set `max_attempts` to 1 by default (#79095)
* Unified Alerting: Set `max_attempts` to 1 by default
The retry logic for unified alerting has been broken as far as v9.4.x, rather than fixing it in one go and causing a headache to our users with rules putting extra load on their datasources - I think a better approach is to simply set 1 as a default and then let our users change it.
I see two cons with this approach:
- Configuration for legacy to unified alerting cannot be ported over automatically, users will have to manually set `max_attempts` to 3 when migrating.
- Users expecting to get any sort of retrying (as with legacy alerting) will not have it out of the box and will have to manually edit the configuration.
Signed-off-by: gotjosh <josue.abreu@gmail.com>
---------
Signed-off-by: gotjosh <josue.abreu@gmail.com>
(cherry picked from commit 0c9356a3c7)
Co-authored-by: gotjosh <josue.abreu@gmail.com>
Alerting: Only warm alert state cache if execute_alerts=true. (#78895)
* Alerting: Only warm alert state cache if execute_alerts=true.
If the Grafana instance is not executing alerts, then Warm()-ing the state
manager is wasteful and could lead to misleading rule status queries, as the
status returned will be always based on the state loaded from the database at
startup, and not the most recent evaluation state.
* Move Warm() down to shared conditional.
(cherry picked from commit 520c927931)
Co-authored-by: Steve Simpson <steve.simpson@grafana.com>
* CI: Update CI/CD tooling and pipelines from main (#76814)
* CI: Update CI/CD tooling and pipelines from main
* Update Makefile
* Comment out validate_openapi_spec_step
* Update broken frontend tests
* Fix validate-npm-packages regex to work without suffix
* Fix cypress image version
(cherry picked from commit 03ecb1db39)
* Comment out kindsysreport/codegen/report.go on Makefile
* Remove modfile step, since modowners doesn't exist
* Disable i18n step
* Prometheus: Handle the response with different field key order (#74567)
* Handle the response with different field key order
* More unit tests to cover edge cases
* Cover more edge cases
* make it simpler
* Better test inputs
(cherry picked from commit 3107459e57)
* Fixes
* [LDAP] Disable removed users on login (#74016)
* [LDAP] Disable removed users on login
* Fix tests
* Add test for user disabling
* Add tests for disabling user behind auth proxy
* Linting.
* Rename setup func
* Account for reviews comments
Co-authored-by: Kalle Persson <kalle.persson@grafana.com>
---------
Co-authored-by: Kalle Persson <kalle.persson@grafana.com>
(cherry picked from commit f900098cc9)
* manual backport of #74016
* LDAP: Fix active sync with large quantities of users (#73834)
* Fix middleware test
---------
Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>
Co-authored-by: Gabriel MABILLE <gabriel.mabille@grafana.com>
* SSE: DSNode to update result with names to make each value identifiable by labels (only Graphite and TestData) (#71246)
* introduce a function checkIfSeriesNeedToBeFixed to scan all value fields in the response and provide a function that updates Series so they can be uniquely identifiable. Only Graphite and TestData are checked.
* update `DSNode.Execute` to run this function and provide it to WideToMany
* update WideToMany to run the fix function if it is not nil
Grafana versioning: Remove `pre` suffix from Grafana version (#73357)
* Remove pre suffix from version
* Fix tests
(cherry picked from commit 0aba319ea0)
Co-authored-by: Dimitris Sotirakis <dimitrios.sotirakis@grafana.com>
AzureMonitor: Allow `serviceTags` and `tags` to be empty for trace results (#73145)
Allow serviceTags and tags to be empty
(cherry picked from commit 95546551bc)
Co-authored-by: Andreas Christou <andreas.christou@grafana.com>
Usage stats: Tune collector execution startup and interval (#72688)
* Do not update statistics at service collector startup
* Configurable collector interval
* Introduce initial random delay
* Prevent reporting metrics until the stats have been collected
* Apply suggestion from code review
(cherry picked from commit afb59af79b)
Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
* Alerting: Fix state manager to not keep datasource_uid and ref_id labels in state after Error (#72216)
(cherry picked from commit 78fc3bcdf4)
* use old QueryError
---------
Co-authored-by: Yuri Tseretyan <yuriy.tseretyan@grafana.com>
Chore: Remove topnav feature flag (#72337)
* Remove topnav feature flag
* Allow deprecated flags to be enabled by default
* change topnav feature flag to deprecated instead
* fix lint
(cherry picked from commit 488eac0e49)
* Alerting: Fix contact point testing with secure settings (#72235)
Fixes double encryption of secure settings during contact point testing and removes code duplication
that helped cause the drift between alertmanager and test endpoint. Also adds integration tests to cover
the regression.
Note: provisioningStore is created to remove cycle and the unnecessary dependency.
(cherry picked from commit d31d175109)
* Alerting: Improve performance of matching captures (#71828)
This commit updates eval.go to improve the performance of matching
captures in the general case. In some cases we have reduced the
runtime of the function from 10s of minutes to a couple 100ms.
In the case where no capture matches the exact labels, we revert to
the current subset/superset match, but with a reduced search space
due to grouping captures.
(cherry picked from commit 8dd3eb856d)
* Add label fingerprints from grafana-plugin-sdk-go
* Remove unsafe.StringData as we use Go 1.19
* Fix lint
---------
Co-authored-by: George Robinson <george.robinson@grafana.com>
CloudMonitoring: Correctly set title and text fields for annotations (#71888)
Correctly set title and text fields
(cherry picked from commit c4d6509145)
Co-authored-by: Andreas Christou <andreas.christou@grafana.com>
Alerting: Add tests for matching captures (#71928)
This commit adds tests for matching captures, which we do not have
at present.
(cherry picked from commit f1af0502db)
Co-authored-by: George Robinson <george.robinson@grafana.com>
Alerting: Sort NumberCaptureValues in EvaluationString (#71927)
This commit changes extractEvalString to sort NumberCaptureValues
in ascending order of Var before building the output string. This
means that users will see EvaluationString in a consistent order,
but also make it possible to assert its output in tests.
(cherry picked from commit 89dcaaf049)
Co-authored-by: George Robinson <george.robinson@grafana.com>
AzureMonitor: Set timespan in Logs Portal URL link (#71841)
Set timespan in Logs Portal URL link
(cherry picked from commit 116e971af3)
Co-authored-by: Andreas Christou <andreas.christou@grafana.com>
Auth: Add support for custom signing keys in auth.azure_ad (#71365)
* fallthrough JWKS validation and caching for Azure
* remove unused field
(cherry picked from commit fbfdd6ba32)
Alerting: No longer silence paused alerts during legacy migration (#71596)
* Alerting: No longer silence paused alerts during legacy migration
Now that we migrate paused legacy alerts to paused UA alert rules, we no longer need to silence them.
(cherry picked from commit 8c6cdf51fc)
Co-authored-by: Matthew Jacobson <matthew.jacobson@grafana.com>
Plugins: Only configure plugin proxy transport once (#71735)
only configure plugin proxy transport once
(cherry picked from commit b59ca7fb22)
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
api: ignore /api/gnet proxy when gzipping responses (#71437)
This PR adds /api/gnet to the list of ignored paths in the gzip middleware.
Without this, when gzip is enabled (`server.enable_gzip = true`), responses
from the gnet proxy are double compressed: once by grafana.com and once by
Grafana itself. With this change we only do one round of compression for these
endpoints.
To test this out, try a request like this with `server.enable_gzip = true`
(after setting `GCOM_TOKEN` to a valid grafana.com token; you may need to
change the 'bsull' slug, too):
curl -v --user admin:admin \
-H "X-Api-Key: $GCOM_TOKEN" \
-H 'Accept-Encoding: gzip' \
localhost:3000/api/gnet/instances/bsull/provisioned-plugins/grafana-ml-app | gzip -d
Note that there are two Content-Encoding: gzip headers before this PR, and
the output is still compressed even after the `gzip -d`. After this PR things
look as expected.
(cherry picked from commit 51c94bccd7)
Co-authored-by: Ben Sully <ben.sully@grafana.com>
Alerting: Fix Alertmanager change detection for receivers with secure settings (#71307)
* Alerting: Make ApplyAlertmanagerConfiguration only decrypt/encrypt new/changed secure settings
Previously, ApplyAlertmanagerConfiguration would decrypt and re-encrypt all secure settings. However, this caused re-encrypted secure settings to be included in the raw configuration when applied to the embedded alertmanager, resulting in changes to the hash. Consequently, even if no actual modifications were made, saving any alertmanager configuration triggered an apply/restart and created a new historical entry in the database.
To address the issue, this modifies ApplyAlertmanagerConfiguration, which is called by POST `api/alertmanager/grafana/config/api/v1/alerts`, to decrypt and re-encrypt only new and updated secure settings. Unchanged secure settings are loaded directly from the database without alteration.
We determine whether secure settings have changed based on the following (already in-use) assumption: Only new or updated secure settings are provided via the POST `api/alertmanager/grafana/config/api/v1/alerts` request, while existing unchanged settings are omitted.
* Ensure saving a grafana-managed contact point will only send new/changed secure settings
Previously, when saving a grafana-managed contact point, empty string values were transmitted for all unset secure settings. This led to potential backend issues, as it assumed that only newly added or updated secure settings would be provided.
To address this, we now exclude empty ('', null, undefined) secure settings, unless there was a pre-existing entry in secureFields for that specific setting. In essence, this means we only transmit an empty secure setting if a previously configured value was cleared.
* Fix linting
* refactor omitEmptyUnlessExisting
* fixup
---------
Co-authored-by: Gilles De Mey <gilles.de.mey@gmail.com>
(cherry picked from commit e3787de470)
Co-authored-by: Matthew Jacobson <matthew.jacobson@grafana.com>
Elasticsearch: Fix using multiple indexes with comma separated string (#71284)
* Revert "Elasticsearch: Use array of strings as index in backend queries (#67276)"
This reverts commit d0ced39847.
* updated tests
(cherry picked from commit e17540bdcd)
Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
Fix: Change getExistingDashboardByTitleAndFolder to get dashboard by title, not slug (#70723)
* Fix: Change getExistingDashboardByTitleAndFolder to get dashboard by title, not slug
* test: add tests for get dashboard with existing name, get dashboard with non existing name, get dashboard with existing name in a folder
* Update pkg/services/dashboards/database/database_test.go
Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
* require specific error for Should be able to get dashboard with existing name
* Update pkg/services/dashboards/database/database_test.go
Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
* implement sofia suggestions to check for specific err, remove logs
* give test more specific name
* implement daniel suggestion of formatting documentation comment in safe way
* fix test title to refer to root directory not specific folder
---------
Co-authored-by: Sofia Papagiannaki <1632407+papagian@users.noreply.github.com>
(cherry picked from commit 67cdae4b7d)
Co-authored-by: Kat Yang <69819079+yangkb09@users.noreply.github.com>
Alexander Weaver
Ashley Harrison
Victor Marin
Karl Persson
gotjosh
grafana-delivery-bot[bot]
Guilherme Caulada
Jack Baldry
Andreas Christou
Horst Gutmann
Yuri Tseretyan
ismail simsek
lean.dev
Sofia Papagiannaki
Josh Hunt
Matthew Jacobson
Dimitris Sotirakis
Jo
Andrew Hackmann
Santiago