Commit Graph

201 Commits

Author SHA1 Message Date
owensmallwood
d715bda8af Unified Storage: Adds pruner to kv backend (#110549)
* WIP adding pruner to kv store impl

* pruner only keeps 20 most recent versions

* ignore grafana-kv-data folder

* extracts some stuff to pruner.go file. Adds tests. Adds kvBackendOptions.

* update logging, comments, exports kvbackendoptions fields

* update nooppruner ref

* fixes field casing in test

* fix test

* linter fixes

* remove comment

* make KvStorageBackend private

* Adds pruner key validation and tests. Fixes broken tests.

* update error message when validating pruner key
2025-09-05 10:02:11 -06:00
Serge Zaitsev
cdd7a2cfd2 Chore: Disable CGo in tests (#108764)
* make cgo optional for sqlite

* update go.mod; check error code differently

* reduce api surface even more

* move test errors into sqlite package

* CGO_ENABLED=0 in unit tests

* disable for enterprise, too

* add driver name constant

* remove unused constants

* make test an integration one

* try integration tests without cgo

* implement error codes for modernc sqlite driver

* typo fix

* missing return

* use error pointer as an interface

* alias the driver

* update workspace, check for test errors too

* check error properly

* add missing driver after rebase

* fix missing import after rebase

* debugging, lets try again

* properly parse options, revert many previous changes

* remove another log

* better url parsing

* revert test rename, leave it for later

* revert reusedSession in unistore

* revert more code

* remove driver name

* revert formatting

* add integration test without cgo for sqlite

* remove tracing and logging

* bring driver alias back

* fix type

* wrong package
2025-09-02 17:24:30 +02:00
Peter Štibraný
da43e2ae07 Don't use transaction in ListModifiedSince. (#110392)
* Don't use transaction in ListModifiedSince.

To guarantee that we don't include events with RV > LatestRV, we include the check in SQL query instead.

* Fix integration test by converting SQL comments into template comments.
2025-09-01 11:39:02 +02:00
Peter Štibraný
238f121e10 LastModifiedSince: return latestRV even when it hasn't changed. (#110391)
Return latestRV.
2025-09-01 08:45:40 +00:00
Peter Štibraný
3a3ba483b1 unified-storage: Skip query when ListModifiedSince cannot return anything. (#110338)
* Skip query when ListModifiedSince cannot return anything.

* Only save listRV if it's different than sinceRV. This saves a disk access if not needed.

* Add test for ListModifiedSince with same RV.
2025-08-29 14:49:20 +02:00
owensmallwood
b4ed217656 Unified Storage: Removes check for name ordering and adds test (#110183)
removes check for name ordering. Adds test to assert expected name and rv ordering.
2025-08-26 12:39:30 -06:00
owensmallwood
bacb5c576c Unified Storage: Fix yield statement (#110134)
fix yield
2025-08-25 20:06:24 -06:00
Stephanie Hingtgen
1091054c25 Unistore: Wire up inline secure values (#110072)
* Unistore: Wire up inline secure values

* add validation and test

* linter
2025-08-25 02:48:49 -06:00
owensmallwood
cace999671 Unified Storage: Add ListSinceModified to StorageBackend (#109697)
* WIP added ListSinceModified to StorageBackend interface

* fix compile time check

* Fix method name

* Fix naming

* fix the rest of the ListSinceModified names

* Uses resource key without name field

* get latest rv from resource_version. Update test.

* adds moar tests

* adds method stub for ListModifiedSince to other StorageBackend implementations

* adds dummy impl to noop storage backend for ListModifiedSince

* skip tests for badger kv backend for now

* fixes tests and adds badgerkv impl for ListModifiedSince

* add badger kv impl

* adds test for new query

* adds test data for new query

* adds ListModifiedSince stub to mockStorageBackend

* uncomment tests

* refactors ListModifiedSince to return an iter.seq2 and handles deduplication. Updates tests. Updates query result sorting.

* remove comments

* remove folder from query (dont need it, yet?)

* regen test queries

* updates test

* updates function comment

* use resourcepb.ResourceKey instead of ModifiedResourceKey

* wrap seq in single transaction. Rollback transaction after 30s if iterator never used. Only track last seen event. Formatting.

* skip TestListModifiedSince for kv backend

* use WatchEvent_Type for action type

* remove redundant fields from order by clause and regen test data for query

* remove redundant fields from order by clause and regen test data for query
2025-08-20 11:54:31 -06:00
Peter Štibraný
c508b01cc5 chore: Close bleve indexes created in temp dir. (#109799)
* Close indexes created in temp dir.
2025-08-18 16:27:57 +02:00
Will Assis
43cda1f1a4 unified-storage: disable watcher when search-after-write feature flag is enabled (#109634)
* disable indexer watcher when search-after-write feature flag is enabled
2025-08-14 10:18:09 -04:00
Mustafa Sencer Özcan
7b4fcd486a Revert "fix: unified resource server list queries order column" (#109529) 2025-08-12 17:29:50 +02:00
Ryan McKinley
e0404f924c K8s/SecureValues: Wire InlineSecureValueSupport to apistore (#109449)
* inline wire

* extra fields

* add variable

* wire
2025-08-11 15:22:56 +03:00
William Wernert
6851d8ae0c Storage: Skip slow test TestIntegrationBenchmarkResourceServer (#109424) 2025-08-08 20:51:12 +00:00
Stephanie Hingtgen
68059344f8 Unified storage: Remove unifiedStorageSearchPermissionFiltering feature flag (#109216)
* Unified storage: Remove unifiedStorageSearchPermissionFiltering ff

* unit test
2025-08-06 02:04:32 -06:00
Serge Zaitsev
6b1143565a Chore: Make cgo optional (for sqlite) (#108756)
* make cgo optional for sqlite

* update go.mod; check error code differently

* reduce api surface even more

* move test errors into sqlite package

* add a comment
2025-07-31 09:25:19 +00:00
Serge Zaitsev
a95fb3a37c Chore: Omit integration tests if short test flag is passed (#108777)
* omit integration tests if short test flag is passed

* Update pkg/services/ngalert/models/receivers_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/services/ngalert/models/receivers_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/cmd/grafana-cli/commands/datamigrations/to_unified_storage_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/services/ngalert/models/receivers_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* fix the rest

* false positive

---------

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-07-28 13:38:54 +02:00
Will Assis
183cd0843e keep instance in the ring on shutdown (#108609) 2025-07-24 15:39:12 -04:00
Will Assis
981fdb29d4 update storage-api to only build index if it owns the namespace (#108418)
* update storage-api to only build index if it owns the namespace

---------

Co-authored-by: Mustafa Sencer Özcan <mustafasencer.ozcan@grafana.com>
2025-07-23 15:59:24 -04:00
Mustafa Sencer Özcan
4a779c4ccb fix: unified resource server list queries order column (#108094) 2025-07-16 08:09:31 +02:00
Mustafa Sencer Özcan
84a4ed612b fix: re-order db initialization logic for resource server (#108059) 2025-07-14 10:25:42 +00:00
Jean-Philippe Quéméner
602b7826c4 refactor(unified-storage): remove resource_server config option (#107649) 2025-07-11 15:33:51 +02:00
Peter Štibraný
9d2eadcfd2 PoC: replace using testing.Short in storage/unified package with integration test skip method (#107887)
* PoC: check if testing.Short is called from integration tests only.

* Rename helper function.

* Fix logic.

* Remove skipping of integration tests from non-integration tests.

* Remove skipping of integration tests from non-integration tests.

* Fix import.
2025-07-09 15:32:10 +00:00
Ryan McKinley
3f502f305d Chore: Update mocks with recent mockery (#107816) 2025-07-09 09:15:34 +02:00
Mustafa Sencer Özcan
b7153d4d20 fix: grpc resource delete error when qos enabled (#107560) 2025-07-03 12:02:05 +02:00
Peter Štibraný
041c343a86 Unified storage: Respect GF_DATABASE_URL override (#105331)
* Database for unified storage resources now reuses DB code that respects URL override.

Access instrument_queries via section getter.
2025-07-03 09:57:40 +00:00
Peter Štibraný
e076c74869 sqltemplate, dbimpl: Remove single-method function types (#107525)
* Remove dbProviderFunc function.

This removes one extra indirection that made the code bit more difficult to navigate.

* Remove indirection function types implementing single-method interfaces.

This streamlines the code and makes it bit easier to navigate.

* Update pkg/storage/unified/sql/sqltemplate/dialect_mysql.go

Co-authored-by: Mustafa Sencer Özcan <32759850+mustafasencer@users.noreply.github.com>

---------

Co-authored-by: Mustafa Sencer Özcan <32759850+mustafasencer@users.noreply.github.com>
2025-07-03 10:38:12 +02:00
Mustafa Sencer Özcan
3674d952a6 fix: use database section for resource server db engine configuration (#107495) 2025-07-02 09:00:51 +00:00
Jean-Philippe Quéméner
9a92900ef3 fix(unified-storage): remove 's' in config parameter (#107457) 2025-07-01 14:06:21 +02:00
Jean-Philippe Quéméner
d76e55371a fix(unified-storage): use the provided connection config parameters (#107455)
* fix(unified-storage): use the provided connection config parameters

* extend tests

* make update-workspace
2025-07-01 11:25:16 +00:00
Mustafa Sencer Özcan
974a2c47f9 feat(unified-storage): add qos support for the resource server (#105939) 2025-07-01 09:22:55 +00:00
Serge Zaitsev
f66a693438 Chore: Rename integration tests to follow the common convention (#105987)
* automatically rename integration tests to follow the common convention

* name tests differently

* alter column type to bigint

* update another column to bigint

* add another alter

* fix subquery for mysql
2025-06-29 16:56:24 +02:00
Will Assis
930c9b3beb fix: double registration panic in auth fallback (#107330)
fix double registration panic in auth fallback
2025-06-27 19:22:16 +02:00
Peter Štibraný
6f3b619f59 Remove use of prometheus.DefaultRegisterer from ProvideUnifiedStorageGrpcService. (#107315) 2025-06-27 15:58:00 +02:00
Will Assis
f4ee58db50 unified-storage: split resource index client (#106297)
* split resource server and index server grpc connection if defined in config
2025-06-27 08:15:52 -04:00
Arati R.
0982cfd9a0 Unified Storage/Search: Add max count config for indexing (#107255)
* Add max count config for indexing
* Build empty index when max count is exceeded
* Address linting
* Refactor buildIndexes
* Add test for max count threshold
* Update test doc comments
* Refactor TestBuildIndexes_MaxCountThreshold to not use mock framework
* Rename mocks used in TestBuildIndexes_MaxCountThreshold

* Refactor mockResourceIndex

* Test setting of indexing threshold configs

* Tweak comments, log

* Fix logging in buildEmptyIndex

* Export and reuse TestDocumentBuilderSupplier

* Reuse MockResourceIndex
2025-06-27 14:00:39 +02:00
Jean-Philippe Quéméner
5f21f320f7 fix(unified-storage): use continue token containing both formats for dualwriter (#106525) 2025-06-13 15:59:46 +02:00
Stephanie Hingtgen
7864c1660f K8s: Get trash fixes (#106411) 2025-06-11 13:54:38 -05:00
Jean-Philippe Quéméner
1196d66c5e feat(unified-storage): log continue token on error (#106475) 2025-06-10 16:41:39 +02:00
Georges Chaudy
04d39cbbc6 unistore: split ListIterator and ListHistory in StorageBackend (#105654)
split listIterator from ListHistory
2025-05-23 15:00:18 +02:00
Jean-Philippe Quéméner
392c1a71c9 Storage: Take migration_locking setting into account (#105938) 2025-05-23 14:11:21 +02:00
Peter Štibraný
c4d3eb1cd0 Remove support for Google Spanner database. (#105846)
* Remove support for Google Spanner database.
2025-05-23 11:35:59 +02:00
Jean-Philippe Quéméner
a13fe75de8 refactor(unified-storage): set the GUID in the resource server (#105683) 2025-05-21 09:49:49 +02:00
Dana Axinte
a7922912fe SecretsManager: Introduce secrets database wrapper (#105472)
SecretsManager: Introduce secret database wrapper

Co-authored-by: PoorlyDefinedBehaviour <brunotj2015@hotmail.com>
Co-authored-by: Leandro Deveikis <leandro.deveikis@gmail.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-05-20 11:48:47 +01:00
Mustafa Sencer Özcan
91882cf50e chore(unified-storage): add tracing for unified storage folder implementation (#105601) 2025-05-19 23:25:08 +03:00
Will Assis
bdae4424e8 unified-storage: setup distributor module option A (#104737)
* setup distributor module

* move lifecycler into resource server provider

* remove ring/client pool setup from distributor module and use the same ring/client pool between storage server module and distributor module

* implement resourcestore server methods

* make healthcheck fail if ring is not running
2025-05-19 07:46:03 -04:00
Jean-Philippe Quéméner
5d8f2e0ec3 refactor(unified-storage): extract list iterator and add tests (#105600) 2025-05-19 13:05:49 +02:00
Peter Štibraný
1ffe653156 Revert change of order of checks when setting up resource DB (#105514)
* Revert change of order in newResourceDBProvider

* Ignore error in provisioningSecrets when working with wrong DB.

* Update workspace.

* Add owner.
2025-05-16 11:35:36 +03:00
Jean-Philippe Quéméner
002f46736a refactor(unified-storage): move generated protos to own pkg (#105356) 2025-05-15 21:36:52 +02:00
Peter Štibraný
7268d6c1bc admin api: Fix reencryption of private_key in signing_key table, introduce test for reencryption (#105010)
* Add integration test for reencrypting secrets.
Rotate keys before reencryption.
Test for setting table from enterprise repo.
2025-05-15 09:51:25 +02:00