Commit Graph
248 Commits
Author SHA1 Message Date
Rafael Paulovic c8da64e4eb chore: separate resource service into search and storage 2026-01-15 00:49:08 +01:00
Rafael Paulovic 2fab497c18 chore: use only needed methods in storage interface
- continue cleanup and separation
2026-01-14 21:14:24 +01:00
mayorandClaude Opus 4.5 a7aa55f908 Add search-server target and configurable search mode
- Add SearchServer module target for standalone search service
- Add search_mode config: "", "embedded" (default), "remote"
- Add search_server_address config for remote search server
- Create remote_search.go: gRPC client wrapper for remote search
- Create search_service.go: standalone search gRPC service
- Modify service.go: conditional search mode handling
- Backward compatible: empty/embedded mode works as before

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 17:49:03 +01:00
mayorandClaude Opus 4.5 0a61846b5e Initialize backend before search server
The backend's Init() must be called before search.Init() because
buildIndexes() calls storage.GetResourceStats() which requires
the database connection to be established.

Previously, backend.Init() was called by ResourceServer.Init()
through the Lifecycle hooks, but now search.Init() runs before
ResourceServer is created.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 16:58:30 +01:00
mayorandClaude Opus 4.5 785cc739ee Fix nil pointer in NewSearchServer - use serverOptions.Backend
The backend may be created inside the function and assigned to
serverOptions.Backend, not opts.Backend. Using opts.Backend caused
a nil pointer dereference in searchSupport.buildIndexes.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 16:22:42 +01:00
Peter Štibraný c8f1efe7c7 Initial prototype extracting SearchServer from ResourceServer. 2026-01-12 17:50:39 +01:00
Will Assis f669bc4448 unified-storage: refactor Sql backend and sqlkv compat tests (#115849)
* move sql and sqlkv backends compatibility tests

* refactor compatibility tests

* run storage backend tests with and without rvmanager

* fix

* fix

* fmt

* fix

* address feedback

* fmt
2026-01-08 15:06:44 -05:00
Will Assis 58e9e4a56d unified-storage: fixes for sqlkv to work with postgres (#115961)
* unified-storage: fixes for sqlkv to work with postgres
2026-01-08 08:21:35 -05:00
Stephanie Hingtgen eb2a390425 Unistore: Prevent deadlock on startup errors (#115799) 2026-01-05 00:51:23 -07:00
Will Assis 4164239f56 unified-storage: implement sqlkv Save method (#115458)
* unified-storage: sqlkv save method
2025-12-19 14:27:06 -05:00
Will Assis 99f5f14de7 unified-storage: move rvmanager into its own package (#115445)
* unified-storage: move rvmanager into its own package so it can be reused with sqlkv later
2025-12-18 18:35:32 -05:00
Peter Štibraný 8a160a8ca1 Convert unique keys in 3 tables to primary keys (#115421)
* Added method for adding migrations for convering unique to primary key.

Based on existing migration for `file` table (in `db_file_storage.go`) migrations.

* Added better default migration names. Added ability to override migration name.

* Use ConvertUniqueKeyToPrimaryKey for cloud_migration_snapshot_partition table.

* Convert resource_version UQE to PK.

* Convert secret_encrypted_value UQE to PK.

* Removed extra test.

* Removed testdata.

* Remove support for renaming migrations for now. We can bring it in later, when we want to convert existing migrations for file, file_meta and setting tables.

* Revert removal of ColumnName to ease backporting, since this field is referenced from enterprise code.

* Use quoted identifiers in Postgres statement.
2025-12-17 15:37:49 +01:00
Will Assis 12dd3dffe0 unified-storage: sqlkv skeleton (#115176)
* implement sqlkv skeleton and include sqlkv in badgerkv tests
2025-12-15 08:56:15 -05:00
Will Assis 1b59c82b74 Revert "Unified-storage: sql backend key path backfill (#115033)" (#115257)
This reverts commit b2dd095bd8.
2025-12-12 17:00:08 +00:00
Will Assis b2dd095bd8 Unified-storage: sql backend key path backfill (#115033)
* unified-storage: add migration to backfill key_path in resource_history
2025-12-12 08:09:51 -05:00
Will Assis 30045c02c0 unified-storage: add index on resource_history key_path column (#115113) 2025-12-10 16:03:58 -05:00
Will Assis 755b479be4 unified-storage: make sql backend update key_path for kv store (#114879)
* unified-storage: update resource_history_update_rv.sql to populate key_path in resource_history
2025-12-10 07:06:06 -05:00
owensmallwoodandSteve Simpson a3daf0e39d Unified storage: Add quotas app to apiserver (#114425)
* initial generation

* went through doc to add new resource

* added dummy kind so grafana will run

* added dummy handler and custom route

* fix app name

* gets custom route working - still a dummy route

* adds groupOverride to manifest

* adds quotas to grpc client and server

* WIP - trying to get api recognized - not working

* Gets route working

* fixes group and resource vars

* expects group and resource as separate params

* set content-type header on response

* removes Quotas kind and regens

* Update grafana-app-sdk to v0.48.5

* Update codegen

* updates manifest

* formatting

* updates grafana-app-sdk version to 0.48.5

* regen ResourceClient mocks

* adds tests

* remove commented code

* uncomment go mod tidy

* fix tests and make update workspace

* adds quotas app to codeowners

* formatting

* make gen-apps

* deletes temp file

* fix generated folder code

* make gofmt

* make gen-go

* make update-workspace

* add COPY apps/quotas to Dockerfile

* fix test mock

* fixes undefined NewFolderStatus()

* make gen-apps, and add func for NewFolderStatus

* make gen-apps again

* make update-workspace

* regen folder_object_gen.go

* gofmt

* fix linting

* apps/folder make update-workspace

* make gen-apps

* make gen-apps

* fixes enterprise_imports.go

* go get testcontainers

* adds feature toggle

* make update-workspace

* fix go mod

* fix another client mock

---------

Co-authored-by: Steve Simpson <steve@grafana.com>
2025-12-09 09:40:34 -06:00
Jean-Philippe Quéméner 1f5fd1c0da chore(unified-storage): align how we do tracing (#114998) 2025-12-09 14:53:53 +01:00
Rafael Bortolon Paulovic ff97bfc772 fix(unified): key_path column default (#114859)
fix: key_path column default
2025-12-04 12:40:21 -05:00
Will Assis 994e1dd58f unified-storage: sqlkv migrations (#114790)
* unified-storage: create resource_events table and add key_path column to resource_history

* Update resource_history_insert template

* update test snapshots

* use latin encoding for key_path and bump size to 2048
2025-12-04 09:21:17 -05:00
Will AssisandGeorges Chaudy 047e6d45fa unified-storage: use name instead of offset in kvstore continue token (#113560)
* unified-storage: use name instead of offset in kvstore continue token

---------

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-12-02 10:35:03 -05:00
Rafael Bortolon Paulovic 12c6d7e83f fix(unified): in-proc SQLite data migration (#114537)
* feat: unified storage migrations integration tests

* chore: add comment and adjust db path name

* chore: refactor test cases into interface

* fix: unified SQLite migration with SQLStore migrator

* revert changes to newResourceDBProvider
2025-11-28 13:13:35 +01:00
owensmallwood df2f528612 Unified Storage: Adds overrides service to resource server (#113794)
* first pass of adding quotas service resource server

* passes prom reg as param

init quota service as part of server params

* init quota service as part of server params

* adds config and only creates quota service when overrides file path is defined

* when quota service enabled, check quota on create and log result

* update log message

* adds tests for quota service

* adds tests for config reloading when the file changes

* fix linter errors

* fix comment

* use startAndAwaitRunning

* Simplifies quotas service. Call manager.GetConfig() when getting quota instead of watching for changes.

* adds tracing to quotas service

* adds nsr attributes to traces when getting quotas and resource stats

* update comment

* update comment

remove check for nil overrides since it will (should) never happen

* fix linter error

* refactors naming to overrides service

checks quotas in separate function

* fix quotas naming

* fixes more quotas -> overrides naming

* use logger from ctx

* linter - remove trailing whitespace

* log FromContext() when checking quotas

* adds events to spans instead of create new spans

updates tenant -> namespace naming

few other minor fixes
2025-11-27 10:29:16 -06:00
João Calisto c2c443757d Unified Storage: allow rebuilding indexes for resource with a new grpc endpoint (#113748)
* Unified Storage: allow rebuilding indexes for resource from a new grpc endpoint

* remove log line

* fix trace def

* lint

* fix after rebase

* addressing code review changes

* update with one channel per rebuild request

* other review suggestions

* update with review suggestions

* run mockery generate for MockResourceClient

* update tests

* update tests and lint

* fix test
2025-11-21 16:42:15 +00:00
owensmallwood 8dddff3ce4 Unified Storage: Pass ns, group, resource to GetResourceStats instead of just namespace (#114050)
* passes nsr to GetResourceStats instead of just namespace

* removes ns check

* fixes failing tests

* make update-workspace

* pass group and resource from rebuild request when getting resource stats
2025-11-18 13:05:21 -06:00
Rafael Bortolon Paulovic 32da63a20f chore: remove US history pruner feature toggle (#114014) 2025-11-17 20:47:37 +01:00
Jean-Philippe Quéméner 589435b7c2 fix(unified-storage): resource server tracing (#113582) 2025-11-07 11:51:32 +01:00
Denis VodopianovandDave Henderson 81683d554d chore : Deprecating FeatureToggles.IsEnabledGlobally (#112885)
* add deprecation on featuremgmt.IsEnabledGlobally

* add nolint reason

* add reasonable deprecation message

* remove junk edits

* add more nolints

* addressing review comments

* Update pkg/services/featuremgmt/models.go

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>

---------

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>
2025-10-24 12:02:53 -04:00
Peter Štibraný a4aa3529c8 Cleanup old entries from resource_last_import_time table. (#112438)
* Cleanup old entries from resource_last_import_time table.

* Add index for last_import_time column.

* Address review feedback.
2025-10-23 11:17:08 +00:00
maicon 3d112755de unistore: add timeout config for resource server joining the ring (#112392)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-10-14 15:27:08 -04:00
Georges Chaudy 616de7b566 Fix kvstore wire (#111827) 2025-10-14 11:30:07 +02:00
Ryan McKinley 6af50482a1 Chore: CleanupTestDB in unifed storage tests (#112223) 2025-10-09 22:22:59 +03:00
Peter Štibraný d801b87db9 LastImportTime for resource. (#112153)
* LastImportTime for resource.

* Make StorageBackendImpl implement GetResourceLastImportTimes

* More missing implementations of GetResourceLastImportTimes

* Fix import.

* Skip TestGetResourceLastImportTime in TestBadgerKVStorageBackend.

* Implement GetResourceLastImportTimes by mockStorageBackend

* Bump test tolerance.

* Fix postgres query and timezone.

* Fix postgres query and timezone.

* Make linter happy.
2025-10-09 11:27:11 +02:00
Will Assis b26647e042 unified-storage: setup rollout-operator endpoint (#111768)
* implement endpoint so that storage/search api can prepare for downscale if enabled
2025-10-01 16:34:30 -04:00
Ryan McKinley c530cacb1c Storage: Propagate RV to server on update+delete (#111866) 2025-10-01 18:00:02 +03:00
Peter Štibraný 357aa7d314 Improve search index eviction (#111542)
* Modify index eviction mechanism such that unowned indexes are also evicted.

* Propagate OwnsIndex function to bleve backend

Fix tests.

Stop eviction goroutine when stopping backend.

Make linter happy.

Make sure we stop backend created by tests.

Review suggestion.

Removed newline.
2025-09-24 16:54:35 +02:00
Will Assis 33ff6dbb9e unified-storage: add feature flag to use ngram for indexing (#111265)
* unified-storage: add feature flag to use ngram instead of edge-ngram for indexing
2025-09-24 10:36:50 +02:00
Peter ŠtibranýandStephanie Hingtgen f062e5a85f Stop background tasks and when stopping bleve backend. (#111249)
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-09-23 18:04:49 +02:00
Mustafa Sencer Özcan 4c2240dcc3 fix: introduce an init lock for the storage backend integration tests (#111166) 2025-09-16 14:18:06 +00:00
Alexander ZobninandJo Garnier 294fd943c0 Chore: Update authlib (#110880)
* Chore: Update authlib

* exclude incompatible version of github.com/grafana/gomemcache

* Update go-jose to v4

* fix jose imports

* remove jose v3 from go.mod

* fix tests

* fix serialize

* fix failing live tests

* add v1 of ES256 testkeys. Port tests to use ES256 instead of HS256

* accept more signature algs for okta and azuread

* azure social graph token sig

* accept more signature algs for oauth refresh and jwt auth

* update workspace

* add a static signer for inproc

* rebase and fix ext_jwt

* fix jwt tests

* apply alex patch on gomemcache

* update linting

* fix ext_jwt panic

* update workspaces

---------

Co-authored-by: Jo Garnier <git@jguer.space>
2025-09-15 12:45:15 +02:00
Mustafa Sencer Özcan 2bd76d4044 fix: bump context deadline for integration tests while backend initialization (#110947) 2025-09-11 17:25:13 +02:00
Peter Štibraný c6c8c6e928 Remove feature toggles from bleve backend. (#110951)
* Remove feature toggles from search backend.

* Remove extra import.

* Remove unnecessary code.
2025-09-11 16:08:41 +02:00
Peter Štibraný 6fa6a5708a Cleanup of old search functionality (#110861)
* Remove support for initMinSize.
Remove support for searchAfterWrite option, now it defaults to true.

* Remove reference to deprecated feature toggle.

* Remove feature toggle completely.

* Remove code related to indexing on watch events.

* Fix compilation error.

* Remove unused field.
2025-09-11 08:23:03 +00:00
Mustafa Sencer Özcan b8b85fbf47 fix: add intrumentation for auth server grpc client (#110875) 2025-09-10 17:33:21 +02:00
Peter Štibraný 7fd9ab9481 Replace check for integration tests. (#110707)
* Replace check for integration tests.
* Revert changes in pkg/tsdb/mysql packages.
* Fix formatting of few tests.
2025-09-08 15:49:49 +02:00
Peter Štibraný d09708fe55 Move SkipIntegrationTestInShortMode to testutil. (#110750)
* Move SkipIntegrationTestInShortMode to testutil.

* make update-workspace
2025-09-08 12:50:31 +02:00
owensmallwood d715bda8af Unified Storage: Adds pruner to kv backend (#110549)
* WIP adding pruner to kv store impl

* pruner only keeps 20 most recent versions

* ignore grafana-kv-data folder

* extracts some stuff to pruner.go file. Adds tests. Adds kvBackendOptions.

* update logging, comments, exports kvbackendoptions fields

* update nooppruner ref

* fixes field casing in test

* fix test

* linter fixes

* remove comment

* make KvStorageBackend private

* Adds pruner key validation and tests. Fixes broken tests.

* update error message when validating pruner key
2025-09-05 10:02:11 -06:00
Serge Zaitsev cdd7a2cfd2 Chore: Disable CGo in tests (#108764)
* make cgo optional for sqlite

* update go.mod; check error code differently

* reduce api surface even more

* move test errors into sqlite package

* CGO_ENABLED=0 in unit tests

* disable for enterprise, too

* add driver name constant

* remove unused constants

* make test an integration one

* try integration tests without cgo

* implement error codes for modernc sqlite driver

* typo fix

* missing return

* use error pointer as an interface

* alias the driver

* update workspace, check for test errors too

* check error properly

* add missing driver after rebase

* fix missing import after rebase

* debugging, lets try again

* properly parse options, revert many previous changes

* remove another log

* better url parsing

* revert test rename, leave it for later

* revert reusedSession in unistore

* revert more code

* remove driver name

* revert formatting

* add integration test without cgo for sqlite

* remove tracing and logging

* bring driver alias back

* fix type

* wrong package
2025-09-02 17:24:30 +02:00
Peter Štibraný da43e2ae07 Don't use transaction in ListModifiedSince. (#110392)
* Don't use transaction in ListModifiedSince.

To guarantee that we don't include events with RV > LatestRV, we include the check in SQL query instead.

* Fix integration test by converting SQL comments into template comments.
2025-09-01 11:39:02 +02:00