Commit Graph
847 Commits
Author SHA1 Message Date
Rafael Paulovic c8da64e4eb chore: separate resource service into search and storage 2026-01-15 00:49:08 +01:00
Rafael Paulovic 2fab497c18 chore: use only needed methods in storage interface
- continue cleanup and separation
2026-01-14 21:14:24 +01:00
mayorandClaude Opus 4.5 a7aa55f908 Add search-server target and configurable search mode
- Add SearchServer module target for standalone search service
- Add search_mode config: "", "embedded" (default), "remote"
- Add search_server_address config for remote search server
- Create remote_search.go: gRPC client wrapper for remote search
- Create search_service.go: standalone search gRPC service
- Modify service.go: conditional search mode handling
- Backward compatible: empty/embedded mode works as before

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 17:49:03 +01:00
mayorandClaude Opus 4.5 0a61846b5e Initialize backend before search server
The backend's Init() must be called before search.Init() because
buildIndexes() calls storage.GetResourceStats() which requires
the database connection to be established.

Previously, backend.Init() was called by ResourceServer.Init()
through the Lifecycle hooks, but now search.Init() runs before
ResourceServer is created.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 16:58:30 +01:00
mayorandClaude Opus 4.5 785cc739ee Fix nil pointer in NewSearchServer - use serverOptions.Backend
The backend may be created inside the function and assigned to
serverOptions.Backend, not opts.Backend. Using opts.Backend caused
a nil pointer dereference in searchSupport.buildIndexes.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-14 16:22:42 +01:00
mayorandClaude Opus 4.5 4913baaf04 Fix compilation errors from SearchServer extraction
- Update NewLocalResourceClient to accept SearchServer parameter
- Add SearchClient methods to ResourceClient interface (client needs both)
- Fix directResourceClient to implement new interface with stub methods
- Update search_and_storage.go test to create SearchServer separately

This continues the work of extracting SearchServer from ResourceServer
to enable independent usage by storage.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-12 17:59:46 +01:00
Peter Štibraný c8f1efe7c7 Initial prototype extracting SearchServer from ResourceServer. 2026-01-12 17:50:39 +01:00
Rafael Bortolon Paulovic 0d7f46c08a chore(unified): remove unifiedStorageSearch feature toggle (#116109) 2026-01-12 13:22:48 +01:00
Ryan McKinley 1b52718c23 Search: include panel titles and types in index (#115742) 2026-01-12 13:21:03 +01:00
Will Assis ba196958cd unified-storage: integration test to CRUD sql backend and sqlkv at the same time (#115907)
* add tests to check that we can CRUD resources to both backends at the same time
2026-01-09 15:38:13 -05:00
Matheus Macabu 52cd096d92 Secrets: Propagate ctx cancel and label it on decryptions (#116058) 2026-01-09 14:25:29 +01:00
Renato Costa ccdafc3fb2 unified-storage: fix event persistence when sqlkv is enabled (#116033) 2026-01-09 07:58:58 -05:00
Rafael Bortolon Paulovic 1c5caeb987 fix(unified): err on timeout to open index (#115953)
* fix(unified): default index path to ephemeral storage mount path

Signed-off-by: Rafael Paulovic <rafael.paulovic@grafana.com>

* Revert "fix: use memory index if index file already open (#115720)"

This reverts commit dc4c106e91.

* fix(unified): set index_path for tests

* chore(unified): re-add bolt open timeout and test for error handling

* chore(unified): return err on timeout

* chore(unified): revert changes to default, use DataPath if index_path not set

* chore(unified): add defaults.ini entry for unified_storage

This is needed to override using env. vars

* chore: revert unrelated diff

* chore: address code review comments

- reduce bolt timeout to 1s
- remove errIndexLocked err type
- add more information about index_path in defaults.ini

---------

Signed-off-by: Rafael Paulovic <rafael.paulovic@grafana.com>
2026-01-09 13:48:54 +01:00
Will Assis f028b9dbdb unified-storage: Sql kv compat tests fields check (#115891)
* add tests to check that resource_history, resource and resource_version are being populated properly with sqlkv
2026-01-08 15:58:32 -05:00
Will Assis f669bc4448 unified-storage: refactor Sql backend and sqlkv compat tests (#115849)
* move sql and sqlkv backends compatibility tests

* refactor compatibility tests

* run storage backend tests with and without rvmanager

* fix

* fix

* fmt

* fix

* address feedback

* fmt
2026-01-08 15:06:44 -05:00
Will Assis 58e9e4a56d unified-storage: fixes for sqlkv to work with postgres (#115961)
* unified-storage: fixes for sqlkv to work with postgres
2026-01-08 08:21:35 -05:00
Mustafa Sencer ÖzcanandRafael Paulovic 088bab8b38 feat: enable auto migration based on resource count (#115619)
* feat(unified): migration at startup based on resource count

-- draft

* feat: introduce auto migration enablement for dashboards & folders

* feat: enable auto migration based on threshold

* fix: improve

* fix: pass in the auto migrate per migration definition

* fix: minor

* fix: only use one options

* fix: test

* fix: test

* fix: tests

* fix: simplify configs

* chore: rename

* fix: add integration test

* fix: add integration test

* fix: integration tests

* chore: add comments

* fix: address comment

* fix: address comments

* fix: test and auto migration flow

* fix: test

---------

Co-authored-by: Rafael Paulovic <rafael.paulovic@grafana.com>
2026-01-08 13:30:40 +01:00
owensmallwood ab0b05550f Unified Storag: Fix readme (#115957)
* fix readme

* spelling
2026-01-07 19:35:33 +00:00
Renato Costa 1e8f1f74ea unified-storage: apply backwards compatibility changes outside sqlkv (#115954) 2026-01-07 13:51:15 -05:00
Ezequiel Victorero 41d7213d7e Docs: Update dualwrite ini config (#115934) 2026-01-07 17:58:58 +01:00
Bruno 7698970f22 Secrets: changes to allow a 3rd party keeper / secret references (#115156)
* Secrets: changes to allow a 3rd party keeper / secret references

* fix test

* make gofmt

* lint

* fix tests

* assign aws secrets manager to @grafana/grafana-operator-experience-squad

* rename Keeper.Reference to Keeper.RetrieveReference

* rename ModelSecretsManager to ModelAWSSecretsManager

* validator: ensure that only one of keeper.Spec.Aws.AccessKey or keeper.Spec.Aws.AssumeRole are set

* move secrets manager dep / go mod tidy

* move secrets manager dep

* keeper validator: move 3rd party secret stores validation to their own functions

* add github.com/aws/aws-sdk-go-v2/service/secretsmanager pkg/extensions/enterprise_imports

* make update-workspace

* undo go.mod changes in /apps

* make update-workspace

* fix test

* add github.com/aws/aws-sdk-go-v2/service/secretsmanager to enterprise_imports

* make update-workspace

* gcworker: handle refs

* make update-workspace

* create toggle: FeatureStageExperimental

* allow features.IsEnabled for now

* format
2026-01-06 11:30:04 -03:00
Stephanie Hingtgen eb2a390425 Unistore: Prevent deadlock on startup errors (#115799) 2026-01-05 00:51:23 -07:00
Mustafa Sencer Özcan dc4c106e91 fix: use memory index if index file already open (#115720)
* feat: add lock structure into bleve index files

* fix: another approach

* fix: new check

* fix: build in memory if index file already open

* fix: update workspace

* fix: add test

* refactor: update func signature

* fix: address comments

* fix: make const
2026-01-02 13:51:51 +01:00
Matheus Macabu 552b6aa717 Secrets: Dont update createdBy when updating a secure value (#115760) 2025-12-31 15:01:22 +01:00
Renato Costa 0284d1e669 unified-storage: add UnixTimestamp support to the sqlkv implementation (#115651)
* unified-storage: add `UnixTimestamp` support to sqlkv implementation

* unified-storage: improve tests and enable all of them on sqlkv
2025-12-19 15:35:22 -05:00
Will Assis 4164239f56 unified-storage: implement sqlkv Save method (#115458)
* unified-storage: sqlkv save method
2025-12-19 14:27:06 -05:00
Renato Costa 338ae95ef5 unified-storage: add BatchDelete support to sqlkv implementation (#115573) 2025-12-19 09:15:23 -05:00
Will Assis 99f5f14de7 unified-storage: move rvmanager into its own package (#115445)
* unified-storage: move rvmanager into its own package so it can be reused with sqlkv later
2025-12-18 18:35:32 -05:00
Mihai Doarna 1862e5dac5 IAM: Fix team search for unistore (#115250)
* fix team search for unistore

* fix search in unistore

* remove field prefix when generating the response

* fix unit test

* address feedback
2025-12-18 18:54:55 +02:00
Renato Costa 19f6dbe1bb unified-storage: add BatchGet support to the sqlkv implementation (#115517)
* unified-storage: add `BatchGet` support to the sqlkv implementation

* address comments

* fix linting
2025-12-18 11:21:36 -05:00
Renato Costa 370d5c2dc2 unified-storage: add Keys support to the sqlkv implementation (#115510)
* unified-storage: add `Keys` support to the sqlkv implementation

* add validation for sort option

* Revert sort order validation, assume desc when invalid
2025-12-17 15:03:59 -05:00
Peter Štibraný 8a160a8ca1 Convert unique keys in 3 tables to primary keys (#115421)
* Added method for adding migrations for convering unique to primary key.

Based on existing migration for `file` table (in `db_file_storage.go`) migrations.

* Added better default migration names. Added ability to override migration name.

* Use ConvertUniqueKeyToPrimaryKey for cloud_migration_snapshot_partition table.

* Convert resource_version UQE to PK.

* Convert secret_encrypted_value UQE to PK.

* Removed extra test.

* Removed testdata.

* Remove support for renaming migrations for now. We can bring it in later, when we want to convert existing migrations for file, file_meta and setting tables.

* Revert removal of ColumnName to ease backporting, since this field is referenced from enterprise code.

* Use quoted identifiers in Postgres statement.
2025-12-17 15:37:49 +01:00
Ryan McKinley d02b2a35cd Provisioning: Ignore dashboard change warning after save (#115401) 2025-12-17 10:17:57 +00:00
Rafael Bortolon Paulovic aa3b9dc4da Unified: Run resource data migrations at startup (#114857)
* chore: uncomment unified migration

* chore: adapt and fix tests

* chore: dynamically bump max conns if needed during migration

* chore: copilot suggestions

* chore: pass ctx in RegisterMigration

* chore: make playlists opt-out and dashboards opt-in

* chore: adjust dashboard test

* chore: disable enable log in test

* chore: address review comments

- do not use pointer config
- add migration registry

* chore: more consistent naming

* chore: fix playlist discovery test
2025-12-17 10:09:57 +01:00
Renato Costa c647140475 unified-storage: add Delete support to the sqlkv implementation (#115443) 2025-12-16 16:19:45 -05:00
Renato Costa bd8f0083ce unified-storage: add Get support to the sqlkv implementation (#115382) 2025-12-16 14:32:13 -05:00
Mustafa Sencer Özcan 2d6c1c4e9e docs: add readme for unified storage on-prem migrations (#114397)
* docs: add documentation for unified storage migrations

* docs: move

* docs: rename title

* docs: add docs

* fix: update table

* fix: lint

* docs: add migration table explanation
2025-12-16 08:00:22 +00:00
Will Assis 12dd3dffe0 unified-storage: sqlkv skeleton (#115176)
* implement sqlkv skeleton and include sqlkv in badgerkv tests
2025-12-15 08:56:15 -05:00
Will Assis 1b59c82b74 Revert "Unified-storage: sql backend key path backfill (#115033)" (#115257)
This reverts commit b2dd095bd8.
2025-12-12 17:00:08 +00:00
Will Assis b2dd095bd8 Unified-storage: sql backend key path backfill (#115033)
* unified-storage: add migration to backfill key_path in resource_history
2025-12-12 08:09:51 -05:00
Misi 0c264b7a5f IAM: Add user search endpoint (#114542)
* wip: initial changes, api registration

* wip

* LegacySearch working with sorting

* Revert mapper change for now

* Clean up

* Cleanup, add integration tests

* Improve tests

* OpenAPI def regen

* Use wildcard search, fix lastSeenAt handling, add lastSeenAtAge

* Add missing files

* Fix merge

* Fixes

* Add tests, regen openapi def

* Address feedback

* Address feedback batch 2

* Chores

* regen openapidef

* Address feedback

* Add tests for paging

* gen apis

* Revert go.mod, go.sum. go.work.sum

* Fix + remove extra tracer parameter
2025-12-11 19:54:48 +01:00
Ryan McKinley a6043deb33 UnifiedStorage: Include RV when fieldSelectors are processed in the backend (#115110) 2025-12-11 09:15:01 +03:00
Will Assis 30045c02c0 unified-storage: add index on resource_history key_path column (#115113) 2025-12-10 16:03:58 -05:00
owensmallwood 5e7b900416 Unified Storage: Adds readme for setting up quotas/overrides (#115031)
* adds readme for setting up quotas/overrides

* updates namespace wording

* updates docs

* update test

* Revert "update test"

This reverts commit ad43e355ba.
2025-12-10 09:21:52 -06:00
Will Assis 755b479be4 unified-storage: make sql backend update key_path for kv store (#114879)
* unified-storage: update resource_history_update_rv.sql to populate key_path in resource_history
2025-12-10 07:06:06 -05:00
owensmallwood 6b50e2d730 Unified Storage: Update yaml decoding for quotas to accomodate top-level overrides key (#115049)
* update yaml decoding for quotas to accomodate top-level overrides key

* update test

* fix test indentation
2025-12-09 23:14:22 +00:00
owensmallwood b8acfade21 Unified Storage: Adds debug logs for checking quotas (#115036)
* adds debug logs for checking quotas

* make update-workspace
2025-12-09 18:59:46 +00:00
owensmallwoodandSteve Simpson a3daf0e39d Unified storage: Add quotas app to apiserver (#114425)
* initial generation

* went through doc to add new resource

* added dummy kind so grafana will run

* added dummy handler and custom route

* fix app name

* gets custom route working - still a dummy route

* adds groupOverride to manifest

* adds quotas to grpc client and server

* WIP - trying to get api recognized - not working

* Gets route working

* fixes group and resource vars

* expects group and resource as separate params

* set content-type header on response

* removes Quotas kind and regens

* Update grafana-app-sdk to v0.48.5

* Update codegen

* updates manifest

* formatting

* updates grafana-app-sdk version to 0.48.5

* regen ResourceClient mocks

* adds tests

* remove commented code

* uncomment go mod tidy

* fix tests and make update workspace

* adds quotas app to codeowners

* formatting

* make gen-apps

* deletes temp file

* fix generated folder code

* make gofmt

* make gen-go

* make update-workspace

* add COPY apps/quotas to Dockerfile

* fix test mock

* fixes undefined NewFolderStatus()

* make gen-apps, and add func for NewFolderStatus

* make gen-apps again

* make update-workspace

* regen folder_object_gen.go

* gofmt

* fix linting

* apps/folder make update-workspace

* make gen-apps

* make gen-apps

* fixes enterprise_imports.go

* go get testcontainers

* adds feature toggle

* make update-workspace

* fix go mod

* fix another client mock

---------

Co-authored-by: Steve Simpson <steve@grafana.com>
2025-12-09 09:40:34 -06:00
Jean-Philippe Quéméner 1f5fd1c0da chore(unified-storage): align how we do tracing (#114998) 2025-12-09 14:53:53 +01:00
Alexander Akhmetov c59d5d1c8e Alerting: Store instance annotations in alert rule state (#114975)
Alerting: Store annotations in alert instance state
2025-12-09 13:52:42 +01:00