Commit Graph
123 Commits
Author SHA1 Message Date
Will Assis f669bc4448 unified-storage: refactor Sql backend and sqlkv compat tests (#115849)
* move sql and sqlkv backends compatibility tests

* refactor compatibility tests

* run storage backend tests with and without rvmanager

* fix

* fix

* fmt

* fix

* address feedback

* fmt
2026-01-08 15:06:44 -05:00
Ryan McKinley a6043deb33 UnifiedStorage: Include RV when fieldSelectors are processed in the backend (#115110) 2025-12-11 09:15:01 +03:00
owensmallwood b8acfade21 Unified Storage: Adds debug logs for checking quotas (#115036)
* adds debug logs for checking quotas

* make update-workspace
2025-12-09 18:59:46 +00:00
owensmallwoodandSteve Simpson a3daf0e39d Unified storage: Add quotas app to apiserver (#114425)
* initial generation

* went through doc to add new resource

* added dummy kind so grafana will run

* added dummy handler and custom route

* fix app name

* gets custom route working - still a dummy route

* adds groupOverride to manifest

* adds quotas to grpc client and server

* WIP - trying to get api recognized - not working

* Gets route working

* fixes group and resource vars

* expects group and resource as separate params

* set content-type header on response

* removes Quotas kind and regens

* Update grafana-app-sdk to v0.48.5

* Update codegen

* updates manifest

* formatting

* updates grafana-app-sdk version to 0.48.5

* regen ResourceClient mocks

* adds tests

* remove commented code

* uncomment go mod tidy

* fix tests and make update workspace

* adds quotas app to codeowners

* formatting

* make gen-apps

* deletes temp file

* fix generated folder code

* make gofmt

* make gen-go

* make update-workspace

* add COPY apps/quotas to Dockerfile

* fix test mock

* fixes undefined NewFolderStatus()

* make gen-apps, and add func for NewFolderStatus

* make gen-apps again

* make update-workspace

* regen folder_object_gen.go

* gofmt

* fix linting

* apps/folder make update-workspace

* make gen-apps

* make gen-apps

* fixes enterprise_imports.go

* go get testcontainers

* adds feature toggle

* make update-workspace

* fix go mod

* fix another client mock

---------

Co-authored-by: Steve Simpson <steve@grafana.com>
2025-12-09 09:40:34 -06:00
Jean-Philippe Quéméner 1f5fd1c0da chore(unified-storage): align how we do tracing (#114998) 2025-12-09 14:53:53 +01:00
Jean-Philippe Quéméner 5b9965ee47 fix(unified-storage): check auth before getting value (#114995) 2025-12-09 12:20:37 +01:00
owensmallwood df2f528612 Unified Storage: Adds overrides service to resource server (#113794)
* first pass of adding quotas service resource server

* passes prom reg as param

init quota service as part of server params

* init quota service as part of server params

* adds config and only creates quota service when overrides file path is defined

* when quota service enabled, check quota on create and log result

* update log message

* adds tests for quota service

* adds tests for config reloading when the file changes

* fix linter errors

* fix comment

* use startAndAwaitRunning

* Simplifies quotas service. Call manager.GetConfig() when getting quota instead of watching for changes.

* adds tracing to quotas service

* adds nsr attributes to traces when getting quotas and resource stats

* update comment

* update comment

remove check for nil overrides since it will (should) never happen

* fix linter error

* refactors naming to overrides service

checks quotas in separate function

* fix quotas naming

* fixes more quotas -> overrides naming

* use logger from ctx

* linter - remove trailing whitespace

* log FromContext() when checking quotas

* adds events to spans instead of create new spans

updates tenant -> namespace naming

few other minor fixes
2025-11-27 10:29:16 -06:00
João Calisto c2c443757d Unified Storage: allow rebuilding indexes for resource with a new grpc endpoint (#113748)
* Unified Storage: allow rebuilding indexes for resource from a new grpc endpoint

* remove log line

* fix trace def

* lint

* fix after rebase

* addressing code review changes

* update with one channel per rebuild request

* other review suggestions

* update with review suggestions

* run mockery generate for MockResourceClient

* update tests

* update tests and lint

* fix test
2025-11-21 16:42:15 +00:00
Peter Štibraný def4c644ec Use infra/log instead of slog in pkg/storage/unified (#114292)
Use infra/log instead of slog. Use log.FromContext to add contextual info to log messages.
2025-11-21 14:38:09 +01:00
owensmallwood 8dddff3ce4 Unified Storage: Pass ns, group, resource to GetResourceStats instead of just namespace (#114050)
* passes nsr to GetResourceStats instead of just namespace

* removes ns check

* fixes failing tests

* make update-workspace

* pass group and resource from rebuild request when getting resource stats
2025-11-18 13:05:21 -06:00
Jean-Philippe Quéméner b9e39cdfcc chore(unified-storage): add debug log for read after write (#113747) 2025-11-12 14:39:03 +01:00
Jean-Philippe Quéméner 589435b7c2 fix(unified-storage): resource server tracing (#113582) 2025-11-07 11:51:32 +01:00
Georges Chaudy ddeb970b4d unistore: replace CDK backend with KV store backend (again) (#113184)
* Reapply "unistore: replace CDK backend with KV store backend"" (#113132)

This reverts commit 7127b2538c.

* enable cluster scope
2025-11-04 00:33:36 +00:00
Stephanie Hingtgen 7127b2538c Revert "unistore: replace CDK backend with KV store backend"" (#113132)
Revert "unistore: replace CDK backend with KV store backend" (#112746)"

This reverts commit fe9c21ebf8.
2025-10-28 21:37:51 +00:00
Georges Chaudy fe9c21ebf8 unistore: replace CDK backend with KV store backend" (#112746)
* deprecate the cdk backend in favor of the kv storage backend

* lint

* fix watchtests

* cover limit=0 for ListHistory

* fix rv too large
2025-10-28 10:56:08 +01:00
owensmallwood 1f7f3c9a5a Unified Storage: KVStore use new validation (#110834)
* name field must match either k8s regex or grafana legacy uid regex. Adds tests.

* moves invalid test to being valid

* uses new US naming validation for kv store validation

* fix function name and update key regex

* fix comment

* use correct errs var

* updates kv key tests
2025-10-09 11:25:39 -06:00
Peter Štibraný d801b87db9 LastImportTime for resource. (#112153)
* LastImportTime for resource.

* Make StorageBackendImpl implement GetResourceLastImportTimes

* More missing implementations of GetResourceLastImportTimes

* Fix import.

* Skip TestGetResourceLastImportTime in TestBadgerKVStorageBackend.

* Implement GetResourceLastImportTimes by mockStorageBackend

* Bump test tolerance.

* Fix postgres query and timezone.

* Fix postgres query and timezone.

* Make linter happy.
2025-10-09 11:27:11 +02:00
Peter Štibraný a44af81082 Unified storage search: Introduce min index update interval (#111978)
* Don't update index more often than specified index_min_update_interval.

* Add artificial sleep at the end of write operations.

* Improve test: check for number of update calls, make diff check less flaky.

* Make test less flaky by allowing for higher diff variance.

* Make test less flaky by allowing for expected update calls variance.
2025-10-06 10:02:03 +02:00
Ryan McKinley c530cacb1c Storage: Propagate RV to server on update+delete (#111866) 2025-10-01 18:00:02 +03:00
Ryan McKinley 2f2289f226 Chore: Update authlib (foder as top level argument) (#111800) 2025-10-01 14:40:28 +00:00
Peter Štibraný 707c486a46 Rebuild search indexes asynchronously (#111829)
* Add "debouncer" queue, which can combine incoming elements.

* Rebuild indexes asynchronously.

* Remove duplicate method.

* Fix bleve tests.

* Extracted combineRebuildRequests and added test for it.

* Add TestShouldRebuildIndex

* Added TestFindIndexesForRebuild

* Added TestFindIndexesForRebuild

* Introduce index_rebuild_workers option.

* Add metric for rebuild queue length.

* Add TestRebuildIndexes.

* Fix import.

* Linter, review feedback.
2025-10-01 11:52:09 +02:00
Ryan McKinley cfbf64c3fd Validation: Move validation into apimachinery package (#111736) 2025-09-30 12:59:33 +00:00
Stephanie Hingtgen e01e61a156 Trash: Keep provisioned objects out of the trash (#111714) 2025-09-28 20:33:57 +00:00
maicon 2e734549b5 Revert "Revert "Unistore: Add validation for resource names"" (#111648)
* Revert "Revert "Unistore: Add validation for resource names" (#111408)"

This reverts commit b3ed600bca.


---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-09-28 20:53:07 +03:00
Ryan McKinley bdefea9acf Storage: Fast path when list with metadata.name=XYZ (#111630) 2025-09-25 15:54:38 +00:00
Peter Štibraný 357aa7d314 Improve search index eviction (#111542)
* Modify index eviction mechanism such that unowned indexes are also evicted.

* Propagate OwnsIndex function to bleve backend

Fix tests.

Stop eviction goroutine when stopping backend.

Make linter happy.

Make sure we stop backend created by tests.

Review suggestion.

Removed newline.
2025-09-24 16:54:35 +02:00
maicon b3ed600bca Revert "Unistore: Add validation for resource names" (#111408)
Revert "Unistore: Add validation for resource names (#110990)"

This reverts commit b63e3fd3ae.
2025-09-19 18:03:23 +00:00
maicon b63e3fd3ae Unistore: Add validation for resource names (#110990)
* Unistore: Add validation for resource names

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-09-19 09:37:17 -03:00
Alexander Zobnin 72d212c5f9 Authlib: Update authz client to use zookies (#111291)
* Authlib: Update authz client to use zookies

* fix zookie return

* fix linter
2025-09-18 16:24:22 +02:00
Peter Štibraný 6fa6a5708a Cleanup of old search functionality (#110861)
* Remove support for initMinSize.
Remove support for searchAfterWrite option, now it defaults to true.

* Remove reference to deprecated feature toggle.

* Remove feature toggle completely.

* Remove code related to indexing on watch events.

* Fix compilation error.

* Remove unused field.
2025-09-11 08:23:03 +00:00
Will Assis 005da25698 fix: ListManagedObjects and CountManagedObjects panic when search is not configured (#110726)
fix ListManagedObjects and CountManagedObjects panicking when search index not configured
2025-09-05 20:58:12 +00:00
Todd Treece 9416abc146 Storage: Set default list limit to 500 (#110356) 2025-09-01 11:09:40 +02:00
Mustafa Sencer Özcan 3056924d10 fix: enforce timeout for requests run in qos (#110162) 2025-08-26 18:40:22 +02:00
owensmallwood cace999671 Unified Storage: Add ListSinceModified to StorageBackend (#109697)
* WIP added ListSinceModified to StorageBackend interface

* fix compile time check

* Fix method name

* Fix naming

* fix the rest of the ListSinceModified names

* Uses resource key without name field

* get latest rv from resource_version. Update test.

* adds moar tests

* adds method stub for ListModifiedSince to other StorageBackend implementations

* adds dummy impl to noop storage backend for ListModifiedSince

* skip tests for badger kv backend for now

* fixes tests and adds badgerkv impl for ListModifiedSince

* add badger kv impl

* adds test for new query

* adds test data for new query

* adds ListModifiedSince stub to mockStorageBackend

* uncomment tests

* refactors ListModifiedSince to return an iter.seq2 and handles deduplication. Updates tests. Updates query result sorting.

* remove comments

* remove folder from query (dont need it, yet?)

* regen test queries

* updates test

* updates function comment

* use resourcepb.ResourceKey instead of ModifiedResourceKey

* wrap seq in single transaction. Rollback transaction after 30s if iterator never used. Only track last seen event. Formatting.

* skip TestListModifiedSince for kv backend

* use WatchEvent_Type for action type

* remove redundant fields from order by clause and regen test data for query

* remove redundant fields from order by clause and regen test data for query
2025-08-20 11:54:31 -06:00
Will Assis 43cda1f1a4 unified-storage: disable watcher when search-after-write feature flag is enabled (#109634)
* disable indexer watcher when search-after-write feature flag is enabled
2025-08-14 10:18:09 -04:00
Ryan McKinley d3df5b8ddd Secrets: Manage secure values inside any resource (#107803) 2025-08-14 12:31:24 +00:00
Ryan McKinley e0404f924c K8s/SecureValues: Wire InlineSecureValueSupport to apistore (#109449)
* inline wire

* extra fields

* add variable

* wire
2025-08-11 15:22:56 +03:00
Will AssisandMustafa Sencer Özcan 981fdb29d4 update storage-api to only build index if it owns the namespace (#108418)
* update storage-api to only build index if it owns the namespace

---------

Co-authored-by: Mustafa Sencer Özcan <mustafasencer.ozcan@grafana.com>
2025-07-23 15:59:24 -04:00
Ryan McKinley 9786389ae8 SecureValues: Support inline secure values in GrafanaMetaAccessor (#107996) 2025-07-11 16:47:54 +00:00
Georges Chaudy b6dd08da2f unistore: fix delete and db closed in kv store (#107918)
* fix delete and db closed

* fix tests
2025-07-10 11:34:36 +02:00
Mustafa Sencer Özcan b7153d4d20 fix: grpc resource delete error when qos enabled (#107560) 2025-07-03 12:02:05 +02:00
Mustafa Sencer Özcan 974a2c47f9 feat(unified-storage): add qos support for the resource server (#105939) 2025-07-01 09:22:55 +00:00
Arati R. 0982cfd9a0 Unified Storage/Search: Add max count config for indexing (#107255)
* Add max count config for indexing
* Build empty index when max count is exceeded
* Address linting
* Refactor buildIndexes
* Add test for max count threshold
* Update test doc comments
* Refactor TestBuildIndexes_MaxCountThreshold to not use mock framework
* Rename mocks used in TestBuildIndexes_MaxCountThreshold

* Refactor mockResourceIndex

* Test setting of indexing threshold configs

* Tweak comments, log

* Fix logging in buildEmptyIndex

* Export and reuse TestDocumentBuilderSupplier

* Reuse MockResourceIndex
2025-06-27 14:00:39 +02:00
Georges Chaudy 9062d88ea0 unistore: filter trash requests (#106767)
* deleted by user

* use the correct checker

* add tests

* refactor
2025-06-24 10:17:34 +02:00
Jean-Philippe Quéméner 5f21f320f7 fix(unified-storage): use continue token containing both formats for dualwriter (#106525) 2025-06-13 15:59:46 +02:00
Stephanie Hingtgen 5135d5c87d Unified storage: Reconstruct index in the background every 24h (#106422) 2025-06-12 14:34:48 -05:00
Georges Chaudy 04d39cbbc6 unistore: split ListIterator and ListHistory in StorageBackend (#105654)
split listIterator from ListHistory
2025-05-23 15:00:18 +02:00
Jean-Philippe Quéméner e57be36936 fix(unified-storage): race in resource server watch (#105786) 2025-05-23 09:19:20 +02:00
Jean-Philippe Quéméner a13fe75de8 refactor(unified-storage): set the GUID in the resource server (#105683) 2025-05-21 09:49:49 +02:00
Will Assis bdae4424e8 unified-storage: setup distributor module option A (#104737)
* setup distributor module

* move lifecycler into resource server provider

* remove ring/client pool setup from distributor module and use the same ring/client pool between storage server module and distributor module

* implement resourcestore server methods

* make healthcheck fail if ring is not running
2025-05-19 07:46:03 -04:00