Commit Graph

845 Commits

Author SHA1 Message Date
Will Assis 86652a6515 unified-storage: sql backend key_path backfill (#116142) 2026-01-13 09:35:02 -05:00
Mustafa Sencer Özcan 5219ccddb6 fix: improve resilience for unified storage and search service grpc clients (#116122)
* fix: reliability

* fix: resilience

* fix: add connection backoff

* fix: reduce backoff
2026-01-13 11:42:21 +00:00
Rafael Bortolon Paulovic 98f271f345 chore(unified): remove unifiedStorageSearchSprinkles feature toggle (#116139)
chore: remove unifiedStorageSearchSprinkles feature flag

The feature flag is no longer needed because:
- OSS: usageinsights code doesn't exist in OSS builds
- Enterprise On-Prem: uses local SQL storage when enable_search=true
- Cloud: explicitly configures sprinkles_api_server URL

The sprinkles functionality now works automatically based on:
- enable_search config (enforced true for unified storage mode 5)
- sprinkles_api_server config (empty = local storage, set = remote API)
2026-01-13 11:24:13 +01:00
Will Assis 8c8efd2494 unified-storage: skip sqlkv/sqlbackend compatibility tests in sqlite (#116164) 2026-01-12 16:31:29 -05:00
Will Assis 1263a3d364 unified-storage: HappyPath and notifier tests + couple of bugfixes (#116087)
* unified-storage: couple of bugfixes and enable HappyPath and notifier sqlkv tests
2026-01-12 12:17:41 -05:00
Rafael Bortolon Paulovic 0d7f46c08a chore(unified): remove unifiedStorageSearch feature toggle (#116109) 2026-01-12 13:22:48 +01:00
Ryan McKinley 1b52718c23 Search: include panel titles and types in index (#115742) 2026-01-12 13:21:03 +01:00
Will Assis ba196958cd unified-storage: integration test to CRUD sql backend and sqlkv at the same time (#115907)
* add tests to check that we can CRUD resources to both backends at the same time
2026-01-09 15:38:13 -05:00
Matheus Macabu 52cd096d92 Secrets: Propagate ctx cancel and label it on decryptions (#116058) 2026-01-09 14:25:29 +01:00
Renato Costa ccdafc3fb2 unified-storage: fix event persistence when sqlkv is enabled (#116033) 2026-01-09 07:58:58 -05:00
Rafael Bortolon Paulovic 1c5caeb987 fix(unified): err on timeout to open index (#115953)
* fix(unified): default index path to ephemeral storage mount path

Signed-off-by: Rafael Paulovic <rafael.paulovic@grafana.com>

* Revert "fix: use memory index if index file already open (#115720)"

This reverts commit dc4c106e91.

* fix(unified): set index_path for tests

* chore(unified): re-add bolt open timeout and test for error handling

* chore(unified): return err on timeout

* chore(unified): revert changes to default, use DataPath if index_path not set

* chore(unified): add defaults.ini entry for unified_storage

This is needed to override using env. vars

* chore: revert unrelated diff

* chore: address code review comments

- reduce bolt timeout to 1s
- remove errIndexLocked err type
- add more information about index_path in defaults.ini

---------

Signed-off-by: Rafael Paulovic <rafael.paulovic@grafana.com>
2026-01-09 13:48:54 +01:00
Will Assis f028b9dbdb unified-storage: Sql kv compat tests fields check (#115891)
* add tests to check that resource_history, resource and resource_version are being populated properly with sqlkv
2026-01-08 15:58:32 -05:00
Will Assis f669bc4448 unified-storage: refactor Sql backend and sqlkv compat tests (#115849)
* move sql and sqlkv backends compatibility tests

* refactor compatibility tests

* run storage backend tests with and without rvmanager

* fix

* fix

* fmt

* fix

* address feedback

* fmt
2026-01-08 15:06:44 -05:00
Will Assis 58e9e4a56d unified-storage: fixes for sqlkv to work with postgres (#115961)
* unified-storage: fixes for sqlkv to work with postgres
2026-01-08 08:21:35 -05:00
Mustafa Sencer Özcan 088bab8b38 feat: enable auto migration based on resource count (#115619)
* feat(unified): migration at startup based on resource count

-- draft

* feat: introduce auto migration enablement for dashboards & folders

* feat: enable auto migration based on threshold

* fix: improve

* fix: pass in the auto migrate per migration definition

* fix: minor

* fix: only use one options

* fix: test

* fix: test

* fix: tests

* fix: simplify configs

* chore: rename

* fix: add integration test

* fix: add integration test

* fix: integration tests

* chore: add comments

* fix: address comment

* fix: address comments

* fix: test and auto migration flow

* fix: test

---------

Co-authored-by: Rafael Paulovic <rafael.paulovic@grafana.com>
2026-01-08 13:30:40 +01:00
owensmallwood ab0b05550f Unified Storag: Fix readme (#115957)
* fix readme

* spelling
2026-01-07 19:35:33 +00:00
Renato Costa 1e8f1f74ea unified-storage: apply backwards compatibility changes outside sqlkv (#115954) 2026-01-07 13:51:15 -05:00
Ezequiel Victorero 41d7213d7e Docs: Update dualwrite ini config (#115934) 2026-01-07 17:58:58 +01:00
Bruno 7698970f22 Secrets: changes to allow a 3rd party keeper / secret references (#115156)
* Secrets: changes to allow a 3rd party keeper / secret references

* fix test

* make gofmt

* lint

* fix tests

* assign aws secrets manager to @grafana/grafana-operator-experience-squad

* rename Keeper.Reference to Keeper.RetrieveReference

* rename ModelSecretsManager to ModelAWSSecretsManager

* validator: ensure that only one of keeper.Spec.Aws.AccessKey or keeper.Spec.Aws.AssumeRole are set

* move secrets manager dep / go mod tidy

* move secrets manager dep

* keeper validator: move 3rd party secret stores validation to their own functions

* add github.com/aws/aws-sdk-go-v2/service/secretsmanager pkg/extensions/enterprise_imports

* make update-workspace

* undo go.mod changes in /apps

* make update-workspace

* fix test

* add github.com/aws/aws-sdk-go-v2/service/secretsmanager to enterprise_imports

* make update-workspace

* gcworker: handle refs

* make update-workspace

* create toggle: FeatureStageExperimental

* allow features.IsEnabled for now

* format
2026-01-06 11:30:04 -03:00
Stephanie Hingtgen eb2a390425 Unistore: Prevent deadlock on startup errors (#115799) 2026-01-05 00:51:23 -07:00
Mustafa Sencer Özcan dc4c106e91 fix: use memory index if index file already open (#115720)
* feat: add lock structure into bleve index files

* fix: another approach

* fix: new check

* fix: build in memory if index file already open

* fix: update workspace

* fix: add test

* refactor: update func signature

* fix: address comments

* fix: make const
2026-01-02 13:51:51 +01:00
Matheus Macabu 552b6aa717 Secrets: Dont update createdBy when updating a secure value (#115760) 2025-12-31 15:01:22 +01:00
Renato Costa 0284d1e669 unified-storage: add UnixTimestamp support to the sqlkv implementation (#115651)
* unified-storage: add `UnixTimestamp` support to sqlkv implementation

* unified-storage: improve tests and enable all of them on sqlkv
2025-12-19 15:35:22 -05:00
Will Assis 4164239f56 unified-storage: implement sqlkv Save method (#115458)
* unified-storage: sqlkv save method
2025-12-19 14:27:06 -05:00
Renato Costa 338ae95ef5 unified-storage: add BatchDelete support to sqlkv implementation (#115573) 2025-12-19 09:15:23 -05:00
Will Assis 99f5f14de7 unified-storage: move rvmanager into its own package (#115445)
* unified-storage: move rvmanager into its own package so it can be reused with sqlkv later
2025-12-18 18:35:32 -05:00
Mihai Doarna 1862e5dac5 IAM: Fix team search for unistore (#115250)
* fix team search for unistore

* fix search in unistore

* remove field prefix when generating the response

* fix unit test

* address feedback
2025-12-18 18:54:55 +02:00
Renato Costa 19f6dbe1bb unified-storage: add BatchGet support to the sqlkv implementation (#115517)
* unified-storage: add `BatchGet` support to the sqlkv implementation

* address comments

* fix linting
2025-12-18 11:21:36 -05:00
Renato Costa 370d5c2dc2 unified-storage: add Keys support to the sqlkv implementation (#115510)
* unified-storage: add `Keys` support to the sqlkv implementation

* add validation for sort option

* Revert sort order validation, assume desc when invalid
2025-12-17 15:03:59 -05:00
Peter Štibraný 8a160a8ca1 Convert unique keys in 3 tables to primary keys (#115421)
* Added method for adding migrations for convering unique to primary key.

Based on existing migration for `file` table (in `db_file_storage.go`) migrations.

* Added better default migration names. Added ability to override migration name.

* Use ConvertUniqueKeyToPrimaryKey for cloud_migration_snapshot_partition table.

* Convert resource_version UQE to PK.

* Convert secret_encrypted_value UQE to PK.

* Removed extra test.

* Removed testdata.

* Remove support for renaming migrations for now. We can bring it in later, when we want to convert existing migrations for file, file_meta and setting tables.

* Revert removal of ColumnName to ease backporting, since this field is referenced from enterprise code.

* Use quoted identifiers in Postgres statement.
2025-12-17 15:37:49 +01:00
Ryan McKinley d02b2a35cd Provisioning: Ignore dashboard change warning after save (#115401) 2025-12-17 10:17:57 +00:00
Rafael Bortolon Paulovic aa3b9dc4da Unified: Run resource data migrations at startup (#114857)
* chore: uncomment unified migration

* chore: adapt and fix tests

* chore: dynamically bump max conns if needed during migration

* chore: copilot suggestions

* chore: pass ctx in RegisterMigration

* chore: make playlists opt-out and dashboards opt-in

* chore: adjust dashboard test

* chore: disable enable log in test

* chore: address review comments

- do not use pointer config
- add migration registry

* chore: more consistent naming

* chore: fix playlist discovery test
2025-12-17 10:09:57 +01:00
Renato Costa c647140475 unified-storage: add Delete support to the sqlkv implementation (#115443) 2025-12-16 16:19:45 -05:00
Renato Costa bd8f0083ce unified-storage: add Get support to the sqlkv implementation (#115382) 2025-12-16 14:32:13 -05:00
Mustafa Sencer Özcan 2d6c1c4e9e docs: add readme for unified storage on-prem migrations (#114397)
* docs: add documentation for unified storage migrations

* docs: move

* docs: rename title

* docs: add docs

* fix: update table

* fix: lint

* docs: add migration table explanation
2025-12-16 08:00:22 +00:00
Will Assis 12dd3dffe0 unified-storage: sqlkv skeleton (#115176)
* implement sqlkv skeleton and include sqlkv in badgerkv tests
2025-12-15 08:56:15 -05:00
Will Assis 1b59c82b74 Revert "Unified-storage: sql backend key path backfill (#115033)" (#115257)
This reverts commit b2dd095bd8.
2025-12-12 17:00:08 +00:00
Will Assis b2dd095bd8 Unified-storage: sql backend key path backfill (#115033)
* unified-storage: add migration to backfill key_path in resource_history
2025-12-12 08:09:51 -05:00
Misi 0c264b7a5f IAM: Add user search endpoint (#114542)
* wip: initial changes, api registration

* wip

* LegacySearch working with sorting

* Revert mapper change for now

* Clean up

* Cleanup, add integration tests

* Improve tests

* OpenAPI def regen

* Use wildcard search, fix lastSeenAt handling, add lastSeenAtAge

* Add missing files

* Fix merge

* Fixes

* Add tests, regen openapi def

* Address feedback

* Address feedback batch 2

* Chores

* regen openapidef

* Address feedback

* Add tests for paging

* gen apis

* Revert go.mod, go.sum. go.work.sum

* Fix + remove extra tracer parameter
2025-12-11 19:54:48 +01:00
Ryan McKinley a6043deb33 UnifiedStorage: Include RV when fieldSelectors are processed in the backend (#115110) 2025-12-11 09:15:01 +03:00
Will Assis 30045c02c0 unified-storage: add index on resource_history key_path column (#115113) 2025-12-10 16:03:58 -05:00
owensmallwood 5e7b900416 Unified Storage: Adds readme for setting up quotas/overrides (#115031)
* adds readme for setting up quotas/overrides

* updates namespace wording

* updates docs

* update test

* Revert "update test"

This reverts commit ad43e355ba.
2025-12-10 09:21:52 -06:00
Will Assis 755b479be4 unified-storage: make sql backend update key_path for kv store (#114879)
* unified-storage: update resource_history_update_rv.sql to populate key_path in resource_history
2025-12-10 07:06:06 -05:00
owensmallwood 6b50e2d730 Unified Storage: Update yaml decoding for quotas to accomodate top-level overrides key (#115049)
* update yaml decoding for quotas to accomodate top-level overrides key

* update test

* fix test indentation
2025-12-09 23:14:22 +00:00
owensmallwood b8acfade21 Unified Storage: Adds debug logs for checking quotas (#115036)
* adds debug logs for checking quotas

* make update-workspace
2025-12-09 18:59:46 +00:00
owensmallwood a3daf0e39d Unified storage: Add quotas app to apiserver (#114425)
* initial generation

* went through doc to add new resource

* added dummy kind so grafana will run

* added dummy handler and custom route

* fix app name

* gets custom route working - still a dummy route

* adds groupOverride to manifest

* adds quotas to grpc client and server

* WIP - trying to get api recognized - not working

* Gets route working

* fixes group and resource vars

* expects group and resource as separate params

* set content-type header on response

* removes Quotas kind and regens

* Update grafana-app-sdk to v0.48.5

* Update codegen

* updates manifest

* formatting

* updates grafana-app-sdk version to 0.48.5

* regen ResourceClient mocks

* adds tests

* remove commented code

* uncomment go mod tidy

* fix tests and make update workspace

* adds quotas app to codeowners

* formatting

* make gen-apps

* deletes temp file

* fix generated folder code

* make gofmt

* make gen-go

* make update-workspace

* add COPY apps/quotas to Dockerfile

* fix test mock

* fixes undefined NewFolderStatus()

* make gen-apps, and add func for NewFolderStatus

* make gen-apps again

* make update-workspace

* regen folder_object_gen.go

* gofmt

* fix linting

* apps/folder make update-workspace

* make gen-apps

* make gen-apps

* fixes enterprise_imports.go

* go get testcontainers

* adds feature toggle

* make update-workspace

* fix go mod

* fix another client mock

---------

Co-authored-by: Steve Simpson <steve@grafana.com>
2025-12-09 09:40:34 -06:00
Jean-Philippe Quéméner 1f5fd1c0da chore(unified-storage): align how we do tracing (#114998) 2025-12-09 14:53:53 +01:00
Alexander Akhmetov c59d5d1c8e Alerting: Store instance annotations in alert rule state (#114975)
Alerting: Store annotations in alert instance state
2025-12-09 13:52:42 +01:00
Jean-Philippe Quéméner 5b9965ee47 fix(unified-storage): check auth before getting value (#114995) 2025-12-09 12:20:37 +01:00
Mihai Doarna b2e1b257b3 IAM: Add search for teams in app platform (#113503)
* add legacy search (wip)

* fix search field name

* implement team search endpoint

* generate openapi spec

* generate endpoints for frontend

* minor fixes

* fix issues found while testing

* add more fields to search result

* add basic unit tests

* add more unit tests

* improve getColumns() func in legacy search

* configure search endpoint in team.cue

* add team search handler

* add the searchTeams endpoint to manifest.cue

* make gofmt

* update openapi spec

* generate frontend endpoints

* remove unused field

* move fields defiitions to separate builder

* fix legacy search

* fix unit tests

* fix unit test

* address feedback

* fix unit test

* update openapi specs

* yarn generate-apis

* add missing unit tests
2025-12-08 15:02:59 +02:00