Commit Graph
679 Commits
Author SHA1 Message Date
Georges Chaudy 1162fa5104 kvstore: Add optimistic locking for unified resource storage backend (#113230)
* Add optimistic concurrency

* add optimistic concurrency

* fix test

* nit

* fix tests for sql

* fix tests for sql

* rebase fix

* add one more check

* Implement GetLatestAndPredecessor method in datastore and add corresponding tests. This new functionality retrieves the latest resource version and its immediate predecessor, handling cases for single and non-existent resources. Update WriteEvent to utilize this method for improved optimistic concurrency control.

* Enhance optimistic concurrency control in WriteEvent method. Added checks for concurrent create operations to ensure only one succeeds, preventing race conditions. Updated tests to validate this behavior with multiple concurrent create attempts.

* lint

* Refactor optimistic concurrency check in WriteEvent method. Simplified the logic by removing unnecessary condition for single version existence, ensuring more robust handling of concurrent modifications.
2025-11-14 10:42:39 +01:00
Jean-Philippe Quéméner e92a976de2 fix(unified-storage): process list items concurrently (#113801) 2025-11-13 12:08:14 +01:00
Mustafa Sencer Özcan 6eac95f860 fix: inject index min update interval into resource server (#113816) 2025-11-13 09:27:54 +00:00
Jean-Philippe Quéméner b9e39cdfcc chore(unified-storage): add debug log for read after write (#113747) 2025-11-12 14:39:03 +01:00
Jean-Philippe Quéméner 76ab09a6a2 feat(unified-storage): add tracing to apistore (#113714) 2025-11-12 09:48:56 +01:00
Jean-Philippe Quéméner 589435b7c2 fix(unified-storage): resource server tracing (#113582) 2025-11-07 11:51:32 +01:00
Nathan Vērzemnieks f4b23253b1 DataSources: Update SDKs in support of auth service (#112101)
* DataSources: Update SDKs for auth service

* Fix deprecated methods & types for new arrow-go version
2025-11-07 10:15:27 +01:00
Will Assis c9e4c26c11 unified-storage: add more list pagination tests (#113543)
* unified-storage: add more list pagination tests
2025-11-06 13:36:02 -05:00
Georges Chaudy ddeb970b4d unistore: replace CDK backend with KV store backend (again) (#113184)
* Reapply "unistore: replace CDK backend with KV store backend"" (#113132)

This reverts commit 7127b2538c.

* enable cluster scope
2025-11-04 00:33:36 +00:00
Georges Chaudy 07bf7b2ae1 kvstore: add cluster-scoped resource support (#113183)
kvstore add experimental clusterscope resource
2025-11-03 15:53:59 -08:00
Georges Chaudy 9533cc4dbb kvstore: Fix missing Folder field in listModifiedSinceEventStore data lookup (#113131)
* fix: ListModifiedSince for resources in folders

* fix: ListModifiedSince for resources in folders
2025-10-28 23:18:15 +01:00
Stephanie Hingtgen 7127b2538c Revert "unistore: replace CDK backend with KV store backend"" (#113132)
Revert "unistore: replace CDK backend with KV store backend" (#112746)"

This reverts commit fe9c21ebf8.
2025-10-28 21:37:51 +00:00
Georges Chaudy 147ff4279b kvstore: fix events lookback + startkey (#113092)
* fix snowflakes events

* add tests
2025-10-28 17:25:55 +01:00
Georges Chaudy fe9c21ebf8 unistore: replace CDK backend with KV store backend" (#112746)
* deprecate the cdk backend in favor of the kv storage backend

* lint

* fix watchtests

* cover limit=0 for ListHistory

* fix rv too large
2025-10-28 10:56:08 +01:00
Ryan McKinley d4d8b2562e Chore: Update gocloud.dev, removing opencensus (#113056) 2025-10-28 09:10:37 +00:00
Georges Chaudy bc9540fadb kvstore: use batch delete to cleanup old events (#112737)
* use batchdelete for cleaning up old events

* comment
2025-10-27 12:27:31 +01:00
Denis VodopianovandDave Henderson 81683d554d chore : Deprecating FeatureToggles.IsEnabledGlobally (#112885)
* add deprecation on featuremgmt.IsEnabledGlobally

* add nolint reason

* add reasonable deprecation message

* remove junk edits

* add more nolints

* addressing review comments

* Update pkg/services/featuremgmt/models.go

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>

---------

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>
2025-10-24 12:02:53 -04:00
Peter Štibraný a4aa3529c8 Cleanup old entries from resource_last_import_time table. (#112438)
* Cleanup old entries from resource_last_import_time table.

* Add index for last_import_time column.

* Address review feedback.
2025-10-23 11:17:08 +00:00
ad9d8098ef IAM: Add email, login field validation to User create/update API (#112391)
* wip

* wip

* wip

(cherry picked from commit 8cedf25892)

* Search seems to be working, the validation is still wip

* Use keyword.Name analyzer for Filterable fields

* Only string fields should be indexed with keyword analyzer

* Change search query for email and login fields to use term query
* Remove unnecessary Exact from the resource protobuf definitions

Co-Authored-By: Ryan McKinley <ryantxu@gmail.com>

* Add legacy search support to the API

* Tests for legacy search, validate and integration tests for user

* Lint

* Add snapshot tests to userDocumentBuilder

* Address CodeQL issues

* Improvements, handle Mode2, tests should pass

* Change default limit from 0 to 1 for requests

* Cleanup

* Add fixme

* Update pkg/registry/apis/iam/register.go

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>

* Update pkg/registry/apis/iam/user/legacy_search.go

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-10-23 11:29:02 +02:00
Will Assis 13f44336f2 unified-storage: remove full ngram feature toggle (#111994)
* unified-storage: remove full ngram feature toggle
2025-10-22 10:11:40 -04:00
Peter Štibraný 3391163f9c Check that we see less calls than allowed maximum. (#112785)
Check that we see less calls than allowed maximum.
2025-10-22 08:39:19 +00:00
Gábor Farkas 8ab7a20265 update dependency grafana/grafana-plugin-sdk-go to v0.281.0 (#112726)
* go get github.com/grafana/grafana-plugin-sdk-go@v0.281.0

* make update-workspace

* updated failing unit test

* disable deprecation warnings

* datasources: allow underscore-prefixed local time range

* updated go.work.sum
2025-10-22 09:02:11 +02:00
Georges Chaudy 9a154ac15f kvstore: add batch delete (#112723)
-add batch delete to the grpc
2025-10-21 21:32:03 +02:00
Georges Chaudy eca379e98e kvstore: update storage backend to use batch get (#112704)
* consume batchget

* remove rv

* remove un-necessary limit

* do not pass stop to the iterators

* ensure we return the error
2025-10-21 15:42:07 +00:00
Georges Chaudy 7be30abab0 fix folder missing in the event key (#112698) 2025-10-21 10:40:56 +00:00
Peter Štibraný a3c4f5cb9f Fix flaky TestConcurrentIndexUpdateAndSearchWithIndexMinUpdateInterval test (#112419)
Collect different RVs from update, and check that difference between them is at least 0.9*minInterval.
2025-10-15 11:17:44 +00:00
maicon 3d112755de unistore: add timeout config for resource server joining the ring (#112392)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-10-14 15:27:08 -04:00
Georges Chaudy 3bb75e2a90 kvstore: Add BatchGet (#111594)
* Add BatchGet

* restructure the batch get tests
2025-10-14 17:19:26 +02:00
Georges Chaudy 616de7b566 Fix kvstore wire (#111827) 2025-10-14 11:30:07 +02:00
Ryan McKinley 6af50482a1 Chore: CleanupTestDB in unifed storage tests (#112223) 2025-10-09 22:22:59 +03:00
owensmallwood 1f7f3c9a5a Unified Storage: KVStore use new validation (#110834)
* name field must match either k8s regex or grafana legacy uid regex. Adds tests.

* moves invalid test to being valid

* uses new US naming validation for kv store validation

* fix function name and update key regex

* fix comment

* use correct errs var

* updates kv key tests
2025-10-09 11:25:39 -06:00
Peter Štibraný d61abe95ad unified-storage: Rebuild indexes with recently-imported resources (#112202)
* Use timestamps reported via GetResourceLastImportTimes to trigger index rebuilds.

* Add test for old last import time.

* Don't reindex after bulk-import. It is now done indirectly via LastImportTime on all instances that own the index.
2025-10-09 16:42:02 +02:00
Peter Štibraný d801b87db9 LastImportTime for resource. (#112153)
* LastImportTime for resource.

* Make StorageBackendImpl implement GetResourceLastImportTimes

* More missing implementations of GetResourceLastImportTimes

* Fix import.

* Skip TestGetResourceLastImportTime in TestBadgerKVStorageBackend.

* Implement GetResourceLastImportTimes by mockStorageBackend

* Bump test tolerance.

* Fix postgres query and timezone.

* Fix postgres query and timezone.

* Make linter happy.
2025-10-09 11:27:11 +02:00
Peter Štibraný 3c6768460c Remove unused index_max_batch_size option to bleve backend. (#112171)
* Remove unused index_max_batch_size option to bleve backend.

* Remove batchSize argument.
2025-10-08 17:04:38 +02:00
Jean-Philippe Quéméner 40dde1d4aa feat(grpc): add client keepalive config (#112158) 2025-10-08 13:22:29 +02:00
maicon 2c5ccd3283 Unistore: ignore name when validating collection keys (#112086)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-10-07 06:45:59 +00:00
Peter Štibraný 2d801eed3c Remove unused arguments from ResourceIndex and SearchBackend interfaces. (#112043)
* Remove unused arguments from ResourceIndex and SearchBackend interfaces.
2025-10-06 11:47:52 +02:00
Peter Štibraný a44af81082 Unified storage search: Introduce min index update interval (#111978)
* Don't update index more often than specified index_min_update_interval.

* Add artificial sleep at the end of write operations.

* Improve test: check for number of update calls, make diff check less flaky.

* Make test less flaky by allowing for higher diff variance.

* Make test less flaky by allowing for expected update calls variance.
2025-10-06 10:02:03 +02:00
Stephanie Hingtgen 56b136a8de Permission Creator: Remove user type check (#112024) 2025-10-04 08:30:08 +03:00
Stephanie Hingtgen 7f805d8e1a Unistore: Add comment on how RV checks are working (#111898) 2025-10-01 16:02:45 -05:00
Will Assis b26647e042 unified-storage: setup rollout-operator endpoint (#111768)
* implement endpoint so that storage/search api can prepare for downscale if enabled
2025-10-01 16:34:30 -04:00
Will Assis 9bde3267bb unified-storage: dont index -- Grafana -- datasource dashboard fields (#111774)
* unified-storage: dont index -- Grafana -- datasource dashboard fields
2025-10-01 11:32:56 -04:00
Ryan McKinley c530cacb1c Storage: Propagate RV to server on update+delete (#111866) 2025-10-01 18:00:02 +03:00
Ryan McKinley 2f2289f226 Chore: Update authlib (foder as top level argument) (#111800) 2025-10-01 14:40:28 +00:00
Will Assis 870131a4aa unified-storage: restrict search to title of documents (#111842)
* restrict search to title of documents
2025-10-01 08:06:26 -04:00
Peter Štibraný 707c486a46 Rebuild search indexes asynchronously (#111829)
* Add "debouncer" queue, which can combine incoming elements.

* Rebuild indexes asynchronously.

* Remove duplicate method.

* Fix bleve tests.

* Extracted combineRebuildRequests and added test for it.

* Add TestShouldRebuildIndex

* Added TestFindIndexesForRebuild

* Added TestFindIndexesForRebuild

* Introduce index_rebuild_workers option.

* Add metric for rebuild queue length.

* Add TestRebuildIndexes.

* Fix import.

* Linter, review feedback.
2025-10-01 11:52:09 +02:00
Ryan McKinley cfbf64c3fd Validation: Move validation into apimachinery package (#111736) 2025-09-30 12:59:33 +00:00
Jean-Philippe Quéméner d0e3e91b03 fix(folders): use namespace_uid for alert count (#111773) 2025-09-29 16:12:34 -05:00
Stephanie Hingtgen e01e61a156 Trash: Keep provisioned objects out of the trash (#111714) 2025-09-28 20:33:57 +00:00
maicon 2e734549b5 Revert "Revert "Unistore: Add validation for resource names"" (#111648)
* Revert "Revert "Unistore: Add validation for resource names" (#111408)"

This reverts commit b3ed600bca.


---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-09-28 20:53:07 +03:00