Commit Graph

763 Commits

Author SHA1 Message Date
Georges Chaudy
1162fa5104 kvstore: Add optimistic locking for unified resource storage backend (#113230)
* Add optimistic concurrency

* add optimistic concurrency

* fix test

* nit

* fix tests for sql

* fix tests for sql

* rebase fix

* add one more check

* Implement GetLatestAndPredecessor method in datastore and add corresponding tests. This new functionality retrieves the latest resource version and its immediate predecessor, handling cases for single and non-existent resources. Update WriteEvent to utilize this method for improved optimistic concurrency control.

* Enhance optimistic concurrency control in WriteEvent method. Added checks for concurrent create operations to ensure only one succeeds, preventing race conditions. Updated tests to validate this behavior with multiple concurrent create attempts.

* lint

* Refactor optimistic concurrency check in WriteEvent method. Simplified the logic by removing unnecessary condition for single version existence, ensuring more robust handling of concurrent modifications.
2025-11-14 10:42:39 +01:00
Jean-Philippe Quéméner
e92a976de2 fix(unified-storage): process list items concurrently (#113801) 2025-11-13 12:08:14 +01:00
Mustafa Sencer Özcan
6eac95f860 fix: inject index min update interval into resource server (#113816) 2025-11-13 09:27:54 +00:00
Mike
6c512dabdc Secrets: Fix MariaDB syntax error due to unsupported CTE syntax (#111610) (#113690)
* Secrets: fix MariaDB syntax error due to unsupported CTE syntax (#111610)

* parametrize guid/created columns and re-generate test fixtures

---------

Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
2025-11-12 15:15:43 +01:00
Mustafa Sencer Özcan
735b776edc fix: cleanup legacy resource if it is created in legacy during dual update (#113753) 2025-11-12 15:02:36 +01:00
Jean-Philippe Quéméner
b9e39cdfcc chore(unified-storage): add debug log for read after write (#113747) 2025-11-12 14:39:03 +01:00
Jean-Philippe Quéméner
76ab09a6a2 feat(unified-storage): add tracing to apistore (#113714) 2025-11-12 09:48:56 +01:00
Jean-Philippe Quéméner
589435b7c2 fix(unified-storage): resource server tracing (#113582) 2025-11-07 11:51:32 +01:00
Nathan Vērzemnieks
f4b23253b1 DataSources: Update SDKs in support of auth service (#112101)
* DataSources: Update SDKs for auth service

* Fix deprecated methods & types for new arrow-go version
2025-11-07 10:15:27 +01:00
Will Assis
c9e4c26c11 unified-storage: add more list pagination tests (#113543)
* unified-storage: add more list pagination tests
2025-11-06 13:36:02 -05:00
Jean-Philippe Quéméner
fd14d4a5ed feat(unified-storage): add tracing to dual writer and legacy storage (#113504)
Co-authored-by: Mustafa Sencer Özcan <32759850+mustafasencer@users.noreply.github.com>
2025-11-06 11:42:46 +00:00
Mustafa Sencer Özcan
51c31e00a4 fix: dual writer log object formatting (#113492)
fix: logging
2025-11-06 11:03:28 +01:00
Rafael Bortolon Paulovic
7281bb7069 fix: background delete on create failure after ctx cancellation (#113442)
* fix: background delete on create failure after ctx cancellation

* fix: address comments

* chore: remove tests using mock
2025-11-06 09:21:11 +01:00
Georges Chaudy
ddeb970b4d unistore: replace CDK backend with KV store backend (again) (#113184)
* Reapply "unistore: replace CDK backend with KV store backend"" (#113132)

This reverts commit 7127b2538c.

* enable cluster scope
2025-11-04 00:33:36 +00:00
Georges Chaudy
07bf7b2ae1 kvstore: add cluster-scoped resource support (#113183)
kvstore add experimental clusterscope resource
2025-11-03 15:53:59 -08:00
Michael Mandrus
cf242def3a Secrets: Refactor data_key_id out of the encoded secure value payload (#112077)
* Secrets: Refactor data_key_id out of the encoded secure value payload (#111852)

* everything compiles

* tests pass

* remove file included by accident

* add entry to gitignore

* some scaffolding for the migration executor

* remove file

* implement and test the migration

* use xkube.Namespace in our interfaces

* add todo

* update wire deps

* add some logs

* fix wire dependency ordering

* create tests to validate error conditions during migrations

* only run the migration as an MT api server

* formatting issues

* change detection of secrets running as MT server

* add todo

* use more specific initializer flags

* make secrets playwright tests work

* set new properties to true by default

* remove developer mode flag

* fix unit tests
2025-10-30 23:04:32 -04:00
Mustafa Sencer Özcan
58098f9339 chore: improve unified dual writer logging (#113203)
chore: improve dual writer logging
2025-10-30 10:12:29 +01:00
Georges Chaudy
9533cc4dbb kvstore: Fix missing Folder field in listModifiedSinceEventStore data lookup (#113131)
* fix: ListModifiedSince for resources in folders

* fix: ListModifiedSince for resources in folders
2025-10-28 23:18:15 +01:00
Stephanie Hingtgen
7127b2538c Revert "unistore: replace CDK backend with KV store backend"" (#113132)
Revert "unistore: replace CDK backend with KV store backend" (#112746)"

This reverts commit fe9c21ebf8.
2025-10-28 21:37:51 +00:00
Georges Chaudy
147ff4279b kvstore: fix events lookback + startkey (#113092)
* fix snowflakes events

* add tests
2025-10-28 17:25:55 +01:00
Georges Chaudy
fe9c21ebf8 unistore: replace CDK backend with KV store backend" (#112746)
* deprecate the cdk backend in favor of the kv storage backend

* lint

* fix watchtests

* cover limit=0 for ListHistory

* fix rv too large
2025-10-28 10:56:08 +01:00
Ryan McKinley
d4d8b2562e Chore: Update gocloud.dev, removing opencensus (#113056) 2025-10-28 09:10:37 +00:00
Georges Chaudy
bc9540fadb kvstore: use batch delete to cleanup old events (#112737)
* use batchdelete for cleaning up old events

* comment
2025-10-27 12:27:31 +01:00
Denis Vodopianov
81683d554d chore : Deprecating FeatureToggles.IsEnabledGlobally (#112885)
* add deprecation on featuremgmt.IsEnabledGlobally

* add nolint reason

* add reasonable deprecation message

* remove junk edits

* add more nolints

* addressing review comments

* Update pkg/services/featuremgmt/models.go

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>

---------

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>
2025-10-24 12:02:53 -04:00
Matheus Macabu
2c6c564c18 Secrets: Add decrypter to decrypt duration metric (#112942) 2025-10-24 15:23:54 +02:00
Peter Štibraný
a4aa3529c8 Cleanup old entries from resource_last_import_time table. (#112438)
* Cleanup old entries from resource_last_import_time table.

* Add index for last_import_time column.

* Address review feedback.
2025-10-23 11:17:08 +00:00
Misi
ad9d8098ef IAM: Add email, login field validation to User create/update API (#112391)
* wip

* wip

* wip

(cherry picked from commit 8cedf25892)

* Search seems to be working, the validation is still wip

* Use keyword.Name analyzer for Filterable fields

* Only string fields should be indexed with keyword analyzer

* Change search query for email and login fields to use term query
* Remove unnecessary Exact from the resource protobuf definitions

Co-Authored-By: Ryan McKinley <ryantxu@gmail.com>

* Add legacy search support to the API

* Tests for legacy search, validate and integration tests for user

* Lint

* Add snapshot tests to userDocumentBuilder

* Address CodeQL issues

* Improvements, handle Mode2, tests should pass

* Change default limit from 0 to 1 for requests

* Cleanup

* Add fixme

* Update pkg/registry/apis/iam/register.go

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>

* Update pkg/registry/apis/iam/user/legacy_search.go

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-10-23 11:29:02 +02:00
Will Assis
13f44336f2 unified-storage: remove full ngram feature toggle (#111994)
* unified-storage: remove full ngram feature toggle
2025-10-22 10:11:40 -04:00
Peter Štibraný
3391163f9c Check that we see less calls than allowed maximum. (#112785)
Check that we see less calls than allowed maximum.
2025-10-22 08:39:19 +00:00
Gábor Farkas
8ab7a20265 update dependency grafana/grafana-plugin-sdk-go to v0.281.0 (#112726)
* go get github.com/grafana/grafana-plugin-sdk-go@v0.281.0

* make update-workspace

* updated failing unit test

* disable deprecation warnings

* datasources: allow underscore-prefixed local time range

* updated go.work.sum
2025-10-22 09:02:11 +02:00
Georges Chaudy
9a154ac15f kvstore: add batch delete (#112723)
-add batch delete to the grpc
2025-10-21 21:32:03 +02:00
Georges Chaudy
eca379e98e kvstore: update storage backend to use batch get (#112704)
* consume batchget

* remove rv

* remove un-necessary limit

* do not pass stop to the iterators

* ensure we return the error
2025-10-21 15:42:07 +00:00
Georges Chaudy
7be30abab0 fix folder missing in the event key (#112698) 2025-10-21 10:40:56 +00:00
Stephanie Hingtgen
891d5f0625 Unistore: Propagate DeprecatedLegacyID on upsert (#112619)
Unistore: Fix label propogation on upsert
2025-10-20 09:29:03 +02:00
Matheus Macabu
8512b219c5 Secrets: Try to get the service identity from the context if not found in errors (#112598) 2025-10-17 17:35:01 +02:00
Ryan McKinley
7c6afeabec Chore: Use IAM DBTime utility in dashboard migrator (#112589) 2025-10-17 17:48:30 +03:00
Peter Štibraný
a3c4f5cb9f Fix flaky TestConcurrentIndexUpdateAndSearchWithIndexMinUpdateInterval test (#112419)
Collect different RVs from update, and check that difference between them is at least 0.9*minInterval.
2025-10-15 11:17:44 +00:00
maicon
3d112755de unistore: add timeout config for resource server joining the ring (#112392)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-10-14 15:27:08 -04:00
Georges Chaudy
3bb75e2a90 kvstore: Add BatchGet (#111594)
* Add BatchGet

* restructure the batch get tests
2025-10-14 17:19:26 +02:00
Georges Chaudy
616de7b566 Fix kvstore wire (#111827) 2025-10-14 11:30:07 +02:00
Ryan McKinley
6af50482a1 Chore: CleanupTestDB in unifed storage tests (#112223) 2025-10-09 22:22:59 +03:00
owensmallwood
1f7f3c9a5a Unified Storage: KVStore use new validation (#110834)
* name field must match either k8s regex or grafana legacy uid regex. Adds tests.

* moves invalid test to being valid

* uses new US naming validation for kv store validation

* fix function name and update key regex

* fix comment

* use correct errs var

* updates kv key tests
2025-10-09 11:25:39 -06:00
Peter Štibraný
d61abe95ad unified-storage: Rebuild indexes with recently-imported resources (#112202)
* Use timestamps reported via GetResourceLastImportTimes to trigger index rebuilds.

* Add test for old last import time.

* Don't reindex after bulk-import. It is now done indirectly via LastImportTime on all instances that own the index.
2025-10-09 16:42:02 +02:00
Peter Štibraný
d801b87db9 LastImportTime for resource. (#112153)
* LastImportTime for resource.

* Make StorageBackendImpl implement GetResourceLastImportTimes

* More missing implementations of GetResourceLastImportTimes

* Fix import.

* Skip TestGetResourceLastImportTime in TestBadgerKVStorageBackend.

* Implement GetResourceLastImportTimes by mockStorageBackend

* Bump test tolerance.

* Fix postgres query and timezone.

* Fix postgres query and timezone.

* Make linter happy.
2025-10-09 11:27:11 +02:00
Peter Štibraný
3c6768460c Remove unused index_max_batch_size option to bleve backend. (#112171)
* Remove unused index_max_batch_size option to bleve backend.

* Remove batchSize argument.
2025-10-08 17:04:38 +02:00
Jean-Philippe Quéméner
40dde1d4aa feat(grpc): add client keepalive config (#112158) 2025-10-08 13:22:29 +02:00
maicon
2c5ccd3283 Unistore: ignore name when validating collection keys (#112086)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-10-07 06:45:59 +00:00
Peter Štibraný
2d801eed3c Remove unused arguments from ResourceIndex and SearchBackend interfaces. (#112043)
* Remove unused arguments from ResourceIndex and SearchBackend interfaces.
2025-10-06 11:47:52 +02:00
Peter Štibraný
a44af81082 Unified storage search: Introduce min index update interval (#111978)
* Don't update index more often than specified index_min_update_interval.

* Add artificial sleep at the end of write operations.

* Improve test: check for number of update calls, make diff check less flaky.

* Make test less flaky by allowing for higher diff variance.

* Make test less flaky by allowing for expected update calls variance.
2025-10-06 10:02:03 +02:00
Stephanie Hingtgen
ce50f2cf61 Revert "Secrets: Refactor data_key_id out of the encoded secure value payload" (#112034)
Revert "Secrets: Refactor data_key_id out of the encoded secure value payload…"

This reverts commit acad92864e.
2025-10-05 09:53:43 +03:00