Commit Graph
455 Commits
Author SHA1 Message Date
Ryan McKinley 040854c8af Search: Allow query field selection (#116238) 2026-01-14 11:55:05 +00:00
Will Assis 1263a3d364 unified-storage: HappyPath and notifier tests + couple of bugfixes (#116087)
* unified-storage: couple of bugfixes and enable HappyPath and notifier sqlkv tests
2026-01-12 12:17:41 -05:00
Rafael Bortolon Paulovic 0d7f46c08a chore(unified): remove unifiedStorageSearch feature toggle (#116109) 2026-01-12 13:22:48 +01:00
Renato Costa ccdafc3fb2 unified-storage: fix event persistence when sqlkv is enabled (#116033) 2026-01-09 07:58:58 -05:00
Will Assis f028b9dbdb unified-storage: Sql kv compat tests fields check (#115891)
* add tests to check that resource_history, resource and resource_version are being populated properly with sqlkv
2026-01-08 15:58:32 -05:00
Will Assis f669bc4448 unified-storage: refactor Sql backend and sqlkv compat tests (#115849)
* move sql and sqlkv backends compatibility tests

* refactor compatibility tests

* run storage backend tests with and without rvmanager

* fix

* fix

* fmt

* fix

* address feedback

* fmt
2026-01-08 15:06:44 -05:00
Will Assis 58e9e4a56d unified-storage: fixes for sqlkv to work with postgres (#115961)
* unified-storage: fixes for sqlkv to work with postgres
2026-01-08 08:21:35 -05:00
Renato Costa 1e8f1f74ea unified-storage: apply backwards compatibility changes outside sqlkv (#115954) 2026-01-07 13:51:15 -05:00
Renato Costa 0284d1e669 unified-storage: add UnixTimestamp support to the sqlkv implementation (#115651)
* unified-storage: add `UnixTimestamp` support to sqlkv implementation

* unified-storage: improve tests and enable all of them on sqlkv
2025-12-19 15:35:22 -05:00
Will Assis 4164239f56 unified-storage: implement sqlkv Save method (#115458)
* unified-storage: sqlkv save method
2025-12-19 14:27:06 -05:00
Renato Costa 338ae95ef5 unified-storage: add BatchDelete support to sqlkv implementation (#115573) 2025-12-19 09:15:23 -05:00
Renato Costa 19f6dbe1bb unified-storage: add BatchGet support to the sqlkv implementation (#115517)
* unified-storage: add `BatchGet` support to the sqlkv implementation

* address comments

* fix linting
2025-12-18 11:21:36 -05:00
Renato Costa 370d5c2dc2 unified-storage: add Keys support to the sqlkv implementation (#115510)
* unified-storage: add `Keys` support to the sqlkv implementation

* add validation for sort option

* Revert sort order validation, assume desc when invalid
2025-12-17 15:03:59 -05:00
Renato Costa c647140475 unified-storage: add Delete support to the sqlkv implementation (#115443) 2025-12-16 16:19:45 -05:00
Renato Costa bd8f0083ce unified-storage: add Get support to the sqlkv implementation (#115382) 2025-12-16 14:32:13 -05:00
Will Assis 12dd3dffe0 unified-storage: sqlkv skeleton (#115176)
* implement sqlkv skeleton and include sqlkv in badgerkv tests
2025-12-15 08:56:15 -05:00
Ryan McKinley a6043deb33 UnifiedStorage: Include RV when fieldSelectors are processed in the backend (#115110) 2025-12-11 09:15:01 +03:00
owensmallwood 6b50e2d730 Unified Storage: Update yaml decoding for quotas to accomodate top-level overrides key (#115049)
* update yaml decoding for quotas to accomodate top-level overrides key

* update test

* fix test indentation
2025-12-09 23:14:22 +00:00
owensmallwood b8acfade21 Unified Storage: Adds debug logs for checking quotas (#115036)
* adds debug logs for checking quotas

* make update-workspace
2025-12-09 18:59:46 +00:00
owensmallwoodandSteve Simpson a3daf0e39d Unified storage: Add quotas app to apiserver (#114425)
* initial generation

* went through doc to add new resource

* added dummy kind so grafana will run

* added dummy handler and custom route

* fix app name

* gets custom route working - still a dummy route

* adds groupOverride to manifest

* adds quotas to grpc client and server

* WIP - trying to get api recognized - not working

* Gets route working

* fixes group and resource vars

* expects group and resource as separate params

* set content-type header on response

* removes Quotas kind and regens

* Update grafana-app-sdk to v0.48.5

* Update codegen

* updates manifest

* formatting

* updates grafana-app-sdk version to 0.48.5

* regen ResourceClient mocks

* adds tests

* remove commented code

* uncomment go mod tidy

* fix tests and make update workspace

* adds quotas app to codeowners

* formatting

* make gen-apps

* deletes temp file

* fix generated folder code

* make gofmt

* make gen-go

* make update-workspace

* add COPY apps/quotas to Dockerfile

* fix test mock

* fixes undefined NewFolderStatus()

* make gen-apps, and add func for NewFolderStatus

* make gen-apps again

* make update-workspace

* regen folder_object_gen.go

* gofmt

* fix linting

* apps/folder make update-workspace

* make gen-apps

* make gen-apps

* fixes enterprise_imports.go

* go get testcontainers

* adds feature toggle

* make update-workspace

* fix go mod

* fix another client mock

---------

Co-authored-by: Steve Simpson <steve@grafana.com>
2025-12-09 09:40:34 -06:00
Jean-Philippe Quéméner 1f5fd1c0da chore(unified-storage): align how we do tracing (#114998) 2025-12-09 14:53:53 +01:00
Jean-Philippe Quéméner 5b9965ee47 fix(unified-storage): check auth before getting value (#114995) 2025-12-09 12:20:37 +01:00
Will Assis ae4d2324d6 Unified storage kvstore bulk import support (#113791)
* implement batchdelete in datastore

* implement bulkprocess in kv storage_backend

* convert bulkRVs to snowflake
2025-12-04 09:26:50 -05:00
Will AssisandGeorges Chaudy 047e6d45fa unified-storage: use name instead of offset in kvstore continue token (#113560)
* unified-storage: use name instead of offset in kvstore continue token

---------

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-12-02 10:35:03 -05:00
Rafael Bortolon Paulovic 12c6d7e83f fix(unified): in-proc SQLite data migration (#114537)
* feat: unified storage migrations integration tests

* chore: add comment and adjust db path name

* chore: refactor test cases into interface

* fix: unified SQLite migration with SQLStore migrator

* revert changes to newResourceDBProvider
2025-11-28 13:13:35 +01:00
Georges Chaudy 5626dc50f8 feat(unified-storage): Add adaptive backoff to event notifier polling (#114401)
* use exponential backoff in notifier

* Enhance BadgerDB configuration in REST options with memory table size and number of memtables

* Enhance BadgerDB configuration in REST options by adding value threshold for LSM vs value log storage
2025-11-27 18:42:01 +01:00
owensmallwood df2f528612 Unified Storage: Adds overrides service to resource server (#113794)
* first pass of adding quotas service resource server

* passes prom reg as param

init quota service as part of server params

* init quota service as part of server params

* adds config and only creates quota service when overrides file path is defined

* when quota service enabled, check quota on create and log result

* update log message

* adds tests for quota service

* adds tests for config reloading when the file changes

* fix linter errors

* fix comment

* use startAndAwaitRunning

* Simplifies quotas service. Call manager.GetConfig() when getting quota instead of watching for changes.

* adds tracing to quotas service

* adds nsr attributes to traces when getting quotas and resource stats

* update comment

* update comment

remove check for nil overrides since it will (should) never happen

* fix linter error

* refactors naming to overrides service

checks quotas in separate function

* fix quotas naming

* fixes more quotas -> overrides naming

* use logger from ctx

* linter - remove trailing whitespace

* log FromContext() when checking quotas

* adds events to spans instead of create new spans

updates tenant -> namespace naming

few other minor fixes
2025-11-27 10:29:16 -06:00
João Calisto c2c443757d Unified Storage: allow rebuilding indexes for resource with a new grpc endpoint (#113748)
* Unified Storage: allow rebuilding indexes for resource from a new grpc endpoint

* remove log line

* fix trace def

* lint

* fix after rebase

* addressing code review changes

* update with one channel per rebuild request

* other review suggestions

* update with review suggestions

* run mockery generate for MockResourceClient

* update tests

* update tests and lint

* fix test
2025-11-21 16:42:15 +00:00
Peter Štibraný def4c644ec Use infra/log instead of slog in pkg/storage/unified (#114292)
Use infra/log instead of slog. Use log.FromContext to add contextual info to log messages.
2025-11-21 14:38:09 +01:00
Peter Štibraný 42db6324c9 Collect and log search stats. (#114107)
* Collect and log search stats.

* Fix compilation problems.
2025-11-19 16:52:47 +01:00
owensmallwood 8dddff3ce4 Unified Storage: Pass ns, group, resource to GetResourceStats instead of just namespace (#114050)
* passes nsr to GetResourceStats instead of just namespace

* removes ns check

* fixes failing tests

* make update-workspace

* pass group and resource from rebuild request when getting resource stats
2025-11-18 13:05:21 -06:00
Georges Chaudy 1162fa5104 kvstore: Add optimistic locking for unified resource storage backend (#113230)
* Add optimistic concurrency

* add optimistic concurrency

* fix test

* nit

* fix tests for sql

* fix tests for sql

* rebase fix

* add one more check

* Implement GetLatestAndPredecessor method in datastore and add corresponding tests. This new functionality retrieves the latest resource version and its immediate predecessor, handling cases for single and non-existent resources. Update WriteEvent to utilize this method for improved optimistic concurrency control.

* Enhance optimistic concurrency control in WriteEvent method. Added checks for concurrent create operations to ensure only one succeeds, preventing race conditions. Updated tests to validate this behavior with multiple concurrent create attempts.

* lint

* Refactor optimistic concurrency check in WriteEvent method. Simplified the logic by removing unnecessary condition for single version existence, ensuring more robust handling of concurrent modifications.
2025-11-14 10:42:39 +01:00
Jean-Philippe Quéméner b9e39cdfcc chore(unified-storage): add debug log for read after write (#113747) 2025-11-12 14:39:03 +01:00
Jean-Philippe Quéméner 589435b7c2 fix(unified-storage): resource server tracing (#113582) 2025-11-07 11:51:32 +01:00
Will Assis c9e4c26c11 unified-storage: add more list pagination tests (#113543)
* unified-storage: add more list pagination tests
2025-11-06 13:36:02 -05:00
Georges Chaudy ddeb970b4d unistore: replace CDK backend with KV store backend (again) (#113184)
* Reapply "unistore: replace CDK backend with KV store backend"" (#113132)

This reverts commit 7127b2538c.

* enable cluster scope
2025-11-04 00:33:36 +00:00
Georges Chaudy 07bf7b2ae1 kvstore: add cluster-scoped resource support (#113183)
kvstore add experimental clusterscope resource
2025-11-03 15:53:59 -08:00
Georges Chaudy 9533cc4dbb kvstore: Fix missing Folder field in listModifiedSinceEventStore data lookup (#113131)
* fix: ListModifiedSince for resources in folders

* fix: ListModifiedSince for resources in folders
2025-10-28 23:18:15 +01:00
Stephanie Hingtgen 7127b2538c Revert "unistore: replace CDK backend with KV store backend"" (#113132)
Revert "unistore: replace CDK backend with KV store backend" (#112746)"

This reverts commit fe9c21ebf8.
2025-10-28 21:37:51 +00:00
Georges Chaudy 147ff4279b kvstore: fix events lookback + startkey (#113092)
* fix snowflakes events

* add tests
2025-10-28 17:25:55 +01:00
Georges Chaudy fe9c21ebf8 unistore: replace CDK backend with KV store backend" (#112746)
* deprecate the cdk backend in favor of the kv storage backend

* lint

* fix watchtests

* cover limit=0 for ListHistory

* fix rv too large
2025-10-28 10:56:08 +01:00
Georges Chaudy bc9540fadb kvstore: use batch delete to cleanup old events (#112737)
* use batchdelete for cleaning up old events

* comment
2025-10-27 12:27:31 +01:00
Denis VodopianovandDave Henderson 81683d554d chore : Deprecating FeatureToggles.IsEnabledGlobally (#112885)
* add deprecation on featuremgmt.IsEnabledGlobally

* add nolint reason

* add reasonable deprecation message

* remove junk edits

* add more nolints

* addressing review comments

* Update pkg/services/featuremgmt/models.go

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>

---------

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>
2025-10-24 12:02:53 -04:00
Georges Chaudy 9a154ac15f kvstore: add batch delete (#112723)
-add batch delete to the grpc
2025-10-21 21:32:03 +02:00
Georges Chaudy eca379e98e kvstore: update storage backend to use batch get (#112704)
* consume batchget

* remove rv

* remove un-necessary limit

* do not pass stop to the iterators

* ensure we return the error
2025-10-21 15:42:07 +00:00
Georges Chaudy 7be30abab0 fix folder missing in the event key (#112698) 2025-10-21 10:40:56 +00:00
Georges Chaudy 3bb75e2a90 kvstore: Add BatchGet (#111594)
* Add BatchGet

* restructure the batch get tests
2025-10-14 17:19:26 +02:00
Georges Chaudy 616de7b566 Fix kvstore wire (#111827) 2025-10-14 11:30:07 +02:00
owensmallwood 1f7f3c9a5a Unified Storage: KVStore use new validation (#110834)
* name field must match either k8s regex or grafana legacy uid regex. Adds tests.

* moves invalid test to being valid

* uses new US naming validation for kv store validation

* fix function name and update key regex

* fix comment

* use correct errs var

* updates kv key tests
2025-10-09 11:25:39 -06:00
Peter Štibraný d61abe95ad unified-storage: Rebuild indexes with recently-imported resources (#112202)
* Use timestamps reported via GetResourceLastImportTimes to trigger index rebuilds.

* Add test for old last import time.

* Don't reindex after bulk-import. It is now done indirectly via LastImportTime on all instances that own the index.
2025-10-09 16:42:02 +02:00