Commit Graph

232 Commits

Author SHA1 Message Date
Will Assis
755b479be4 unified-storage: make sql backend update key_path for kv store (#114879)
* unified-storage: update resource_history_update_rv.sql to populate key_path in resource_history
2025-12-10 07:06:06 -05:00
owensmallwood
a3daf0e39d Unified storage: Add quotas app to apiserver (#114425)
* initial generation

* went through doc to add new resource

* added dummy kind so grafana will run

* added dummy handler and custom route

* fix app name

* gets custom route working - still a dummy route

* adds groupOverride to manifest

* adds quotas to grpc client and server

* WIP - trying to get api recognized - not working

* Gets route working

* fixes group and resource vars

* expects group and resource as separate params

* set content-type header on response

* removes Quotas kind and regens

* Update grafana-app-sdk to v0.48.5

* Update codegen

* updates manifest

* formatting

* updates grafana-app-sdk version to 0.48.5

* regen ResourceClient mocks

* adds tests

* remove commented code

* uncomment go mod tidy

* fix tests and make update workspace

* adds quotas app to codeowners

* formatting

* make gen-apps

* deletes temp file

* fix generated folder code

* make gofmt

* make gen-go

* make update-workspace

* add COPY apps/quotas to Dockerfile

* fix test mock

* fixes undefined NewFolderStatus()

* make gen-apps, and add func for NewFolderStatus

* make gen-apps again

* make update-workspace

* regen folder_object_gen.go

* gofmt

* fix linting

* apps/folder make update-workspace

* make gen-apps

* make gen-apps

* fixes enterprise_imports.go

* go get testcontainers

* adds feature toggle

* make update-workspace

* fix go mod

* fix another client mock

---------

Co-authored-by: Steve Simpson <steve@grafana.com>
2025-12-09 09:40:34 -06:00
Jean-Philippe Quéméner
1f5fd1c0da chore(unified-storage): align how we do tracing (#114998) 2025-12-09 14:53:53 +01:00
Rafael Bortolon Paulovic
ff97bfc772 fix(unified): key_path column default (#114859)
fix: key_path column default
2025-12-04 12:40:21 -05:00
Will Assis
994e1dd58f unified-storage: sqlkv migrations (#114790)
* unified-storage: create resource_events table and add key_path column to resource_history

* Update resource_history_insert template

* update test snapshots

* use latin encoding for key_path and bump size to 2048
2025-12-04 09:21:17 -05:00
Will Assis
047e6d45fa unified-storage: use name instead of offset in kvstore continue token (#113560)
* unified-storage: use name instead of offset in kvstore continue token

---------

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-12-02 10:35:03 -05:00
Rafael Bortolon Paulovic
12c6d7e83f fix(unified): in-proc SQLite data migration (#114537)
* feat: unified storage migrations integration tests

* chore: add comment and adjust db path name

* chore: refactor test cases into interface

* fix: unified SQLite migration with SQLStore migrator

* revert changes to newResourceDBProvider
2025-11-28 13:13:35 +01:00
owensmallwood
df2f528612 Unified Storage: Adds overrides service to resource server (#113794)
* first pass of adding quotas service resource server

* passes prom reg as param

init quota service as part of server params

* init quota service as part of server params

* adds config and only creates quota service when overrides file path is defined

* when quota service enabled, check quota on create and log result

* update log message

* adds tests for quota service

* adds tests for config reloading when the file changes

* fix linter errors

* fix comment

* use startAndAwaitRunning

* Simplifies quotas service. Call manager.GetConfig() when getting quota instead of watching for changes.

* adds tracing to quotas service

* adds nsr attributes to traces when getting quotas and resource stats

* update comment

* update comment

remove check for nil overrides since it will (should) never happen

* fix linter error

* refactors naming to overrides service

checks quotas in separate function

* fix quotas naming

* fixes more quotas -> overrides naming

* use logger from ctx

* linter - remove trailing whitespace

* log FromContext() when checking quotas

* adds events to spans instead of create new spans

updates tenant -> namespace naming

few other minor fixes
2025-11-27 10:29:16 -06:00
João Calisto
c2c443757d Unified Storage: allow rebuilding indexes for resource with a new grpc endpoint (#113748)
* Unified Storage: allow rebuilding indexes for resource from a new grpc endpoint

* remove log line

* fix trace def

* lint

* fix after rebase

* addressing code review changes

* update with one channel per rebuild request

* other review suggestions

* update with review suggestions

* run mockery generate for MockResourceClient

* update tests

* update tests and lint

* fix test
2025-11-21 16:42:15 +00:00
owensmallwood
8dddff3ce4 Unified Storage: Pass ns, group, resource to GetResourceStats instead of just namespace (#114050)
* passes nsr to GetResourceStats instead of just namespace

* removes ns check

* fixes failing tests

* make update-workspace

* pass group and resource from rebuild request when getting resource stats
2025-11-18 13:05:21 -06:00
Rafael Bortolon Paulovic
32da63a20f chore: remove US history pruner feature toggle (#114014) 2025-11-17 20:47:37 +01:00
Jean-Philippe Quéméner
589435b7c2 fix(unified-storage): resource server tracing (#113582) 2025-11-07 11:51:32 +01:00
Denis Vodopianov
81683d554d chore : Deprecating FeatureToggles.IsEnabledGlobally (#112885)
* add deprecation on featuremgmt.IsEnabledGlobally

* add nolint reason

* add reasonable deprecation message

* remove junk edits

* add more nolints

* addressing review comments

* Update pkg/services/featuremgmt/models.go

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>

---------

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>
2025-10-24 12:02:53 -04:00
Peter Štibraný
a4aa3529c8 Cleanup old entries from resource_last_import_time table. (#112438)
* Cleanup old entries from resource_last_import_time table.

* Add index for last_import_time column.

* Address review feedback.
2025-10-23 11:17:08 +00:00
maicon
3d112755de unistore: add timeout config for resource server joining the ring (#112392)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-10-14 15:27:08 -04:00
Georges Chaudy
616de7b566 Fix kvstore wire (#111827) 2025-10-14 11:30:07 +02:00
Ryan McKinley
6af50482a1 Chore: CleanupTestDB in unifed storage tests (#112223) 2025-10-09 22:22:59 +03:00
Peter Štibraný
d801b87db9 LastImportTime for resource. (#112153)
* LastImportTime for resource.

* Make StorageBackendImpl implement GetResourceLastImportTimes

* More missing implementations of GetResourceLastImportTimes

* Fix import.

* Skip TestGetResourceLastImportTime in TestBadgerKVStorageBackend.

* Implement GetResourceLastImportTimes by mockStorageBackend

* Bump test tolerance.

* Fix postgres query and timezone.

* Fix postgres query and timezone.

* Make linter happy.
2025-10-09 11:27:11 +02:00
Will Assis
b26647e042 unified-storage: setup rollout-operator endpoint (#111768)
* implement endpoint so that storage/search api can prepare for downscale if enabled
2025-10-01 16:34:30 -04:00
Ryan McKinley
c530cacb1c Storage: Propagate RV to server on update+delete (#111866) 2025-10-01 18:00:02 +03:00
Peter Štibraný
357aa7d314 Improve search index eviction (#111542)
* Modify index eviction mechanism such that unowned indexes are also evicted.

* Propagate OwnsIndex function to bleve backend

Fix tests.

Stop eviction goroutine when stopping backend.

Make linter happy.

Make sure we stop backend created by tests.

Review suggestion.

Removed newline.
2025-09-24 16:54:35 +02:00
Will Assis
33ff6dbb9e unified-storage: add feature flag to use ngram for indexing (#111265)
* unified-storage: add feature flag to use ngram instead of edge-ngram for indexing
2025-09-24 10:36:50 +02:00
Peter Štibraný
f062e5a85f Stop background tasks and when stopping bleve backend. (#111249)
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-09-23 18:04:49 +02:00
Mustafa Sencer Özcan
4c2240dcc3 fix: introduce an init lock for the storage backend integration tests (#111166) 2025-09-16 14:18:06 +00:00
Alexander Zobnin
294fd943c0 Chore: Update authlib (#110880)
* Chore: Update authlib

* exclude incompatible version of github.com/grafana/gomemcache

* Update go-jose to v4

* fix jose imports

* remove jose v3 from go.mod

* fix tests

* fix serialize

* fix failing live tests

* add v1 of ES256 testkeys. Port tests to use ES256 instead of HS256

* accept more signature algs for okta and azuread

* azure social graph token sig

* accept more signature algs for oauth refresh and jwt auth

* update workspace

* add a static signer for inproc

* rebase and fix ext_jwt

* fix jwt tests

* apply alex patch on gomemcache

* update linting

* fix ext_jwt panic

* update workspaces

---------

Co-authored-by: Jo Garnier <git@jguer.space>
2025-09-15 12:45:15 +02:00
Mustafa Sencer Özcan
2bd76d4044 fix: bump context deadline for integration tests while backend initialization (#110947) 2025-09-11 17:25:13 +02:00
Peter Štibraný
c6c8c6e928 Remove feature toggles from bleve backend. (#110951)
* Remove feature toggles from search backend.

* Remove extra import.

* Remove unnecessary code.
2025-09-11 16:08:41 +02:00
Peter Štibraný
6fa6a5708a Cleanup of old search functionality (#110861)
* Remove support for initMinSize.
Remove support for searchAfterWrite option, now it defaults to true.

* Remove reference to deprecated feature toggle.

* Remove feature toggle completely.

* Remove code related to indexing on watch events.

* Fix compilation error.

* Remove unused field.
2025-09-11 08:23:03 +00:00
Mustafa Sencer Özcan
b8b85fbf47 fix: add intrumentation for auth server grpc client (#110875) 2025-09-10 17:33:21 +02:00
Peter Štibraný
7fd9ab9481 Replace check for integration tests. (#110707)
* Replace check for integration tests.
* Revert changes in pkg/tsdb/mysql packages.
* Fix formatting of few tests.
2025-09-08 15:49:49 +02:00
Peter Štibraný
d09708fe55 Move SkipIntegrationTestInShortMode to testutil. (#110750)
* Move SkipIntegrationTestInShortMode to testutil.

* make update-workspace
2025-09-08 12:50:31 +02:00
owensmallwood
d715bda8af Unified Storage: Adds pruner to kv backend (#110549)
* WIP adding pruner to kv store impl

* pruner only keeps 20 most recent versions

* ignore grafana-kv-data folder

* extracts some stuff to pruner.go file. Adds tests. Adds kvBackendOptions.

* update logging, comments, exports kvbackendoptions fields

* update nooppruner ref

* fixes field casing in test

* fix test

* linter fixes

* remove comment

* make KvStorageBackend private

* Adds pruner key validation and tests. Fixes broken tests.

* update error message when validating pruner key
2025-09-05 10:02:11 -06:00
Serge Zaitsev
cdd7a2cfd2 Chore: Disable CGo in tests (#108764)
* make cgo optional for sqlite

* update go.mod; check error code differently

* reduce api surface even more

* move test errors into sqlite package

* CGO_ENABLED=0 in unit tests

* disable for enterprise, too

* add driver name constant

* remove unused constants

* make test an integration one

* try integration tests without cgo

* implement error codes for modernc sqlite driver

* typo fix

* missing return

* use error pointer as an interface

* alias the driver

* update workspace, check for test errors too

* check error properly

* add missing driver after rebase

* fix missing import after rebase

* debugging, lets try again

* properly parse options, revert many previous changes

* remove another log

* better url parsing

* revert test rename, leave it for later

* revert reusedSession in unistore

* revert more code

* remove driver name

* revert formatting

* add integration test without cgo for sqlite

* remove tracing and logging

* bring driver alias back

* fix type

* wrong package
2025-09-02 17:24:30 +02:00
Peter Štibraný
da43e2ae07 Don't use transaction in ListModifiedSince. (#110392)
* Don't use transaction in ListModifiedSince.

To guarantee that we don't include events with RV > LatestRV, we include the check in SQL query instead.

* Fix integration test by converting SQL comments into template comments.
2025-09-01 11:39:02 +02:00
Peter Štibraný
238f121e10 LastModifiedSince: return latestRV even when it hasn't changed. (#110391)
Return latestRV.
2025-09-01 08:45:40 +00:00
Peter Štibraný
3a3ba483b1 unified-storage: Skip query when ListModifiedSince cannot return anything. (#110338)
* Skip query when ListModifiedSince cannot return anything.

* Only save listRV if it's different than sinceRV. This saves a disk access if not needed.

* Add test for ListModifiedSince with same RV.
2025-08-29 14:49:20 +02:00
owensmallwood
b4ed217656 Unified Storage: Removes check for name ordering and adds test (#110183)
removes check for name ordering. Adds test to assert expected name and rv ordering.
2025-08-26 12:39:30 -06:00
owensmallwood
bacb5c576c Unified Storage: Fix yield statement (#110134)
fix yield
2025-08-25 20:06:24 -06:00
Stephanie Hingtgen
1091054c25 Unistore: Wire up inline secure values (#110072)
* Unistore: Wire up inline secure values

* add validation and test

* linter
2025-08-25 02:48:49 -06:00
owensmallwood
cace999671 Unified Storage: Add ListSinceModified to StorageBackend (#109697)
* WIP added ListSinceModified to StorageBackend interface

* fix compile time check

* Fix method name

* Fix naming

* fix the rest of the ListSinceModified names

* Uses resource key without name field

* get latest rv from resource_version. Update test.

* adds moar tests

* adds method stub for ListModifiedSince to other StorageBackend implementations

* adds dummy impl to noop storage backend for ListModifiedSince

* skip tests for badger kv backend for now

* fixes tests and adds badgerkv impl for ListModifiedSince

* add badger kv impl

* adds test for new query

* adds test data for new query

* adds ListModifiedSince stub to mockStorageBackend

* uncomment tests

* refactors ListModifiedSince to return an iter.seq2 and handles deduplication. Updates tests. Updates query result sorting.

* remove comments

* remove folder from query (dont need it, yet?)

* regen test queries

* updates test

* updates function comment

* use resourcepb.ResourceKey instead of ModifiedResourceKey

* wrap seq in single transaction. Rollback transaction after 30s if iterator never used. Only track last seen event. Formatting.

* skip TestListModifiedSince for kv backend

* use WatchEvent_Type for action type

* remove redundant fields from order by clause and regen test data for query

* remove redundant fields from order by clause and regen test data for query
2025-08-20 11:54:31 -06:00
Peter Štibraný
c508b01cc5 chore: Close bleve indexes created in temp dir. (#109799)
* Close indexes created in temp dir.
2025-08-18 16:27:57 +02:00
Will Assis
43cda1f1a4 unified-storage: disable watcher when search-after-write feature flag is enabled (#109634)
* disable indexer watcher when search-after-write feature flag is enabled
2025-08-14 10:18:09 -04:00
Mustafa Sencer Özcan
7b4fcd486a Revert "fix: unified resource server list queries order column" (#109529) 2025-08-12 17:29:50 +02:00
Ryan McKinley
e0404f924c K8s/SecureValues: Wire InlineSecureValueSupport to apistore (#109449)
* inline wire

* extra fields

* add variable

* wire
2025-08-11 15:22:56 +03:00
William Wernert
6851d8ae0c Storage: Skip slow test TestIntegrationBenchmarkResourceServer (#109424) 2025-08-08 20:51:12 +00:00
Stephanie Hingtgen
68059344f8 Unified storage: Remove unifiedStorageSearchPermissionFiltering feature flag (#109216)
* Unified storage: Remove unifiedStorageSearchPermissionFiltering ff

* unit test
2025-08-06 02:04:32 -06:00
Serge Zaitsev
6b1143565a Chore: Make cgo optional (for sqlite) (#108756)
* make cgo optional for sqlite

* update go.mod; check error code differently

* reduce api surface even more

* move test errors into sqlite package

* add a comment
2025-07-31 09:25:19 +00:00
Serge Zaitsev
a95fb3a37c Chore: Omit integration tests if short test flag is passed (#108777)
* omit integration tests if short test flag is passed

* Update pkg/services/ngalert/models/receivers_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/services/ngalert/models/receivers_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/cmd/grafana-cli/commands/datamigrations/to_unified_storage_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/services/ngalert/models/receivers_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* fix the rest

* false positive

---------

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-07-28 13:38:54 +02:00
Will Assis
183cd0843e keep instance in the ring on shutdown (#108609) 2025-07-24 15:39:12 -04:00
Will Assis
981fdb29d4 update storage-api to only build index if it owns the namespace (#108418)
* update storage-api to only build index if it owns the namespace

---------

Co-authored-by: Mustafa Sencer Özcan <mustafasencer.ozcan@grafana.com>
2025-07-23 15:59:24 -04:00