Commit Graph

763 Commits

Author SHA1 Message Date
Georges Chaudy 1162fa5104 kvstore: Add optimistic locking for unified resource storage backend (#113230)
* Add optimistic concurrency

* add optimistic concurrency

* fix test

* nit

* fix tests for sql

* fix tests for sql

* rebase fix

* add one more check

* Implement GetLatestAndPredecessor method in datastore and add corresponding tests. This new functionality retrieves the latest resource version and its immediate predecessor, handling cases for single and non-existent resources. Update WriteEvent to utilize this method for improved optimistic concurrency control.

* Enhance optimistic concurrency control in WriteEvent method. Added checks for concurrent create operations to ensure only one succeeds, preventing race conditions. Updated tests to validate this behavior with multiple concurrent create attempts.

* lint

* Refactor optimistic concurrency check in WriteEvent method. Simplified the logic by removing unnecessary condition for single version existence, ensuring more robust handling of concurrent modifications.
2025-11-14 10:42:39 +01:00
Jean-Philippe Quéméner e92a976de2 fix(unified-storage): process list items concurrently (#113801) 2025-11-13 12:08:14 +01:00
Mustafa Sencer Özcan 6eac95f860 fix: inject index min update interval into resource server (#113816) 2025-11-13 09:27:54 +00:00
Mike 6c512dabdc Secrets: Fix MariaDB syntax error due to unsupported CTE syntax (#111610) (#113690)
* Secrets: fix MariaDB syntax error due to unsupported CTE syntax (#111610)

* parametrize guid/created columns and re-generate test fixtures

---------

Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
2025-11-12 15:15:43 +01:00
Mustafa Sencer Özcan 735b776edc fix: cleanup legacy resource if it is created in legacy during dual update (#113753) 2025-11-12 15:02:36 +01:00
Jean-Philippe Quéméner b9e39cdfcc chore(unified-storage): add debug log for read after write (#113747) 2025-11-12 14:39:03 +01:00
Jean-Philippe Quéméner 76ab09a6a2 feat(unified-storage): add tracing to apistore (#113714) 2025-11-12 09:48:56 +01:00
Jean-Philippe Quéméner 589435b7c2 fix(unified-storage): resource server tracing (#113582) 2025-11-07 11:51:32 +01:00
Nathan Vērzemnieks f4b23253b1 DataSources: Update SDKs in support of auth service (#112101)
* DataSources: Update SDKs for auth service

* Fix deprecated methods & types for new arrow-go version
2025-11-07 10:15:27 +01:00
Will Assis c9e4c26c11 unified-storage: add more list pagination tests (#113543)
* unified-storage: add more list pagination tests
2025-11-06 13:36:02 -05:00
Jean-Philippe Quéméner fd14d4a5ed feat(unified-storage): add tracing to dual writer and legacy storage (#113504)
Co-authored-by: Mustafa Sencer Özcan <32759850+mustafasencer@users.noreply.github.com>
2025-11-06 11:42:46 +00:00
Mustafa Sencer Özcan 51c31e00a4 fix: dual writer log object formatting (#113492)
fix: logging
2025-11-06 11:03:28 +01:00
Rafael Bortolon Paulovic 7281bb7069 fix: background delete on create failure after ctx cancellation (#113442)
* fix: background delete on create failure after ctx cancellation

* fix: address comments

* chore: remove tests using mock
2025-11-06 09:21:11 +01:00
Georges Chaudy ddeb970b4d unistore: replace CDK backend with KV store backend (again) (#113184)
* Reapply "unistore: replace CDK backend with KV store backend"" (#113132)

This reverts commit 7127b2538c.

* enable cluster scope
2025-11-04 00:33:36 +00:00
Georges Chaudy 07bf7b2ae1 kvstore: add cluster-scoped resource support (#113183)
kvstore add experimental clusterscope resource
2025-11-03 15:53:59 -08:00
Michael Mandrus cf242def3a Secrets: Refactor data_key_id out of the encoded secure value payload (#112077)
* Secrets: Refactor data_key_id out of the encoded secure value payload (#111852)

* everything compiles

* tests pass

* remove file included by accident

* add entry to gitignore

* some scaffolding for the migration executor

* remove file

* implement and test the migration

* use xkube.Namespace in our interfaces

* add todo

* update wire deps

* add some logs

* fix wire dependency ordering

* create tests to validate error conditions during migrations

* only run the migration as an MT api server

* formatting issues

* change detection of secrets running as MT server

* add todo

* use more specific initializer flags

* make secrets playwright tests work

* set new properties to true by default

* remove developer mode flag

* fix unit tests
2025-10-30 23:04:32 -04:00
Mustafa Sencer Özcan 58098f9339 chore: improve unified dual writer logging (#113203)
chore: improve dual writer logging
2025-10-30 10:12:29 +01:00
Georges Chaudy 9533cc4dbb kvstore: Fix missing Folder field in listModifiedSinceEventStore data lookup (#113131)
* fix: ListModifiedSince for resources in folders

* fix: ListModifiedSince for resources in folders
2025-10-28 23:18:15 +01:00
Stephanie Hingtgen 7127b2538c Revert "unistore: replace CDK backend with KV store backend"" (#113132)
Revert "unistore: replace CDK backend with KV store backend" (#112746)"

This reverts commit fe9c21ebf8.
2025-10-28 21:37:51 +00:00
Georges Chaudy 147ff4279b kvstore: fix events lookback + startkey (#113092)
* fix snowflakes events

* add tests
2025-10-28 17:25:55 +01:00
Georges Chaudy fe9c21ebf8 unistore: replace CDK backend with KV store backend" (#112746)
* deprecate the cdk backend in favor of the kv storage backend

* lint

* fix watchtests

* cover limit=0 for ListHistory

* fix rv too large
2025-10-28 10:56:08 +01:00
Ryan McKinley d4d8b2562e Chore: Update gocloud.dev, removing opencensus (#113056) 2025-10-28 09:10:37 +00:00
Georges Chaudy bc9540fadb kvstore: use batch delete to cleanup old events (#112737)
* use batchdelete for cleaning up old events

* comment
2025-10-27 12:27:31 +01:00
Denis Vodopianov 81683d554d chore : Deprecating FeatureToggles.IsEnabledGlobally (#112885)
* add deprecation on featuremgmt.IsEnabledGlobally

* add nolint reason

* add reasonable deprecation message

* remove junk edits

* add more nolints

* addressing review comments

* Update pkg/services/featuremgmt/models.go

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>

---------

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>
2025-10-24 12:02:53 -04:00
Matheus Macabu 2c6c564c18 Secrets: Add decrypter to decrypt duration metric (#112942) 2025-10-24 15:23:54 +02:00
Peter Štibraný a4aa3529c8 Cleanup old entries from resource_last_import_time table. (#112438)
* Cleanup old entries from resource_last_import_time table.

* Add index for last_import_time column.

* Address review feedback.
2025-10-23 11:17:08 +00:00
Misi ad9d8098ef IAM: Add email, login field validation to User create/update API (#112391)
* wip

* wip

* wip

(cherry picked from commit 8cedf25892)

* Search seems to be working, the validation is still wip

* Use keyword.Name analyzer for Filterable fields

* Only string fields should be indexed with keyword analyzer

* Change search query for email and login fields to use term query
* Remove unnecessary Exact from the resource protobuf definitions

Co-Authored-By: Ryan McKinley <ryantxu@gmail.com>

* Add legacy search support to the API

* Tests for legacy search, validate and integration tests for user

* Lint

* Add snapshot tests to userDocumentBuilder

* Address CodeQL issues

* Improvements, handle Mode2, tests should pass

* Change default limit from 0 to 1 for requests

* Cleanup

* Add fixme

* Update pkg/registry/apis/iam/register.go

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>

* Update pkg/registry/apis/iam/user/legacy_search.go

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-10-23 11:29:02 +02:00
Will Assis 13f44336f2 unified-storage: remove full ngram feature toggle (#111994)
* unified-storage: remove full ngram feature toggle
2025-10-22 10:11:40 -04:00
Peter Štibraný 3391163f9c Check that we see less calls than allowed maximum. (#112785)
Check that we see less calls than allowed maximum.
2025-10-22 08:39:19 +00:00
Gábor Farkas 8ab7a20265 update dependency grafana/grafana-plugin-sdk-go to v0.281.0 (#112726)
* go get github.com/grafana/grafana-plugin-sdk-go@v0.281.0

* make update-workspace

* updated failing unit test

* disable deprecation warnings

* datasources: allow underscore-prefixed local time range

* updated go.work.sum
2025-10-22 09:02:11 +02:00
Georges Chaudy 9a154ac15f kvstore: add batch delete (#112723)
-add batch delete to the grpc
2025-10-21 21:32:03 +02:00
Georges Chaudy eca379e98e kvstore: update storage backend to use batch get (#112704)
* consume batchget

* remove rv

* remove un-necessary limit

* do not pass stop to the iterators

* ensure we return the error
2025-10-21 15:42:07 +00:00
Georges Chaudy 7be30abab0 fix folder missing in the event key (#112698) 2025-10-21 10:40:56 +00:00
Stephanie Hingtgen 891d5f0625 Unistore: Propagate DeprecatedLegacyID on upsert (#112619)
Unistore: Fix label propogation on upsert
2025-10-20 09:29:03 +02:00
Matheus Macabu 8512b219c5 Secrets: Try to get the service identity from the context if not found in errors (#112598) 2025-10-17 17:35:01 +02:00
Ryan McKinley 7c6afeabec Chore: Use IAM DBTime utility in dashboard migrator (#112589) 2025-10-17 17:48:30 +03:00
Peter Štibraný a3c4f5cb9f Fix flaky TestConcurrentIndexUpdateAndSearchWithIndexMinUpdateInterval test (#112419)
Collect different RVs from update, and check that difference between them is at least 0.9*minInterval.
2025-10-15 11:17:44 +00:00
maicon 3d112755de unistore: add timeout config for resource server joining the ring (#112392)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-10-14 15:27:08 -04:00
Georges Chaudy 3bb75e2a90 kvstore: Add BatchGet (#111594)
* Add BatchGet

* restructure the batch get tests
2025-10-14 17:19:26 +02:00
Georges Chaudy 616de7b566 Fix kvstore wire (#111827) 2025-10-14 11:30:07 +02:00
Ryan McKinley 6af50482a1 Chore: CleanupTestDB in unifed storage tests (#112223) 2025-10-09 22:22:59 +03:00
owensmallwood 1f7f3c9a5a Unified Storage: KVStore use new validation (#110834)
* name field must match either k8s regex or grafana legacy uid regex. Adds tests.

* moves invalid test to being valid

* uses new US naming validation for kv store validation

* fix function name and update key regex

* fix comment

* use correct errs var

* updates kv key tests
2025-10-09 11:25:39 -06:00
Peter Štibraný d61abe95ad unified-storage: Rebuild indexes with recently-imported resources (#112202)
* Use timestamps reported via GetResourceLastImportTimes to trigger index rebuilds.

* Add test for old last import time.

* Don't reindex after bulk-import. It is now done indirectly via LastImportTime on all instances that own the index.
2025-10-09 16:42:02 +02:00
Peter Štibraný d801b87db9 LastImportTime for resource. (#112153)
* LastImportTime for resource.

* Make StorageBackendImpl implement GetResourceLastImportTimes

* More missing implementations of GetResourceLastImportTimes

* Fix import.

* Skip TestGetResourceLastImportTime in TestBadgerKVStorageBackend.

* Implement GetResourceLastImportTimes by mockStorageBackend

* Bump test tolerance.

* Fix postgres query and timezone.

* Fix postgres query and timezone.

* Make linter happy.
2025-10-09 11:27:11 +02:00
Peter Štibraný 3c6768460c Remove unused index_max_batch_size option to bleve backend. (#112171)
* Remove unused index_max_batch_size option to bleve backend.

* Remove batchSize argument.
2025-10-08 17:04:38 +02:00
Jean-Philippe Quéméner 40dde1d4aa feat(grpc): add client keepalive config (#112158) 2025-10-08 13:22:29 +02:00
maicon 2c5ccd3283 Unistore: ignore name when validating collection keys (#112086)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-10-07 06:45:59 +00:00
Peter Štibraný 2d801eed3c Remove unused arguments from ResourceIndex and SearchBackend interfaces. (#112043)
* Remove unused arguments from ResourceIndex and SearchBackend interfaces.
2025-10-06 11:47:52 +02:00
Peter Štibraný a44af81082 Unified storage search: Introduce min index update interval (#111978)
* Don't update index more often than specified index_min_update_interval.

* Add artificial sleep at the end of write operations.

* Improve test: check for number of update calls, make diff check less flaky.

* Make test less flaky by allowing for higher diff variance.

* Make test less flaky by allowing for expected update calls variance.
2025-10-06 10:02:03 +02:00
Stephanie Hingtgen ce50f2cf61 Revert "Secrets: Refactor data_key_id out of the encoded secure value payload" (#112034)
Revert "Secrets: Refactor data_key_id out of the encoded secure value payload…"

This reverts commit acad92864e.
2025-10-05 09:53:43 +03:00