Commit Graph
115 Commits
Author SHA1 Message Date
Mustafa Sencer ÖzcanandRafael Paulovic 088bab8b38 feat: enable auto migration based on resource count (#115619)
* feat(unified): migration at startup based on resource count

-- draft

* feat: introduce auto migration enablement for dashboards & folders

* feat: enable auto migration based on threshold

* fix: improve

* fix: pass in the auto migrate per migration definition

* fix: minor

* fix: only use one options

* fix: test

* fix: test

* fix: tests

* fix: simplify configs

* chore: rename

* fix: add integration test

* fix: add integration test

* fix: integration tests

* chore: add comments

* fix: address comment

* fix: address comments

* fix: test and auto migration flow

* fix: test

---------

Co-authored-by: Rafael Paulovic <rafael.paulovic@grafana.com>
2026-01-08 13:30:40 +01:00
Rafael Bortolon Paulovic 90af2c3c3b fix(dashboard): panic on nil logger on dashboard accessor (#115545)
fix(dashboard): fix panic on log
2025-12-18 14:11:47 +01:00
Mustafa Sencer Özcan 40976bb1e4 fix: preserve the order when migrating the playlists (#115485)
fix: preserve the order
2025-12-17 13:48:19 +01:00
Rafael Bortolon Paulovic aa3b9dc4da Unified: Run resource data migrations at startup (#114857)
* chore: uncomment unified migration

* chore: adapt and fix tests

* chore: dynamically bump max conns if needed during migration

* chore: copilot suggestions

* chore: pass ctx in RegisterMigration

* chore: make playlists opt-out and dashboards opt-in

* chore: adjust dashboard test

* chore: disable enable log in test

* chore: address review comments

- do not use pointer config
- add migration registry

* chore: more consistent naming

* chore: fix playlist discovery test
2025-12-17 10:09:57 +01:00
Rafael Bortolon Paulovic 5eae7d4f22 feat: legacy ListIterator with batches (#115038)
* feat: legacy ListIterator with batches

* chore: address code review

* chore: remove nil check in nextBatch

* chore: move close before count check

* chore: add err field to batchingIterator for its own errors

* chore: remove unused import
2025-12-10 16:12:08 +01:00
owensmallwoodandSteve Simpson a3daf0e39d Unified storage: Add quotas app to apiserver (#114425)
* initial generation

* went through doc to add new resource

* added dummy kind so grafana will run

* added dummy handler and custom route

* fix app name

* gets custom route working - still a dummy route

* adds groupOverride to manifest

* adds quotas to grpc client and server

* WIP - trying to get api recognized - not working

* Gets route working

* fixes group and resource vars

* expects group and resource as separate params

* set content-type header on response

* removes Quotas kind and regens

* Update grafana-app-sdk to v0.48.5

* Update codegen

* updates manifest

* formatting

* updates grafana-app-sdk version to 0.48.5

* regen ResourceClient mocks

* adds tests

* remove commented code

* uncomment go mod tidy

* fix tests and make update workspace

* adds quotas app to codeowners

* formatting

* make gen-apps

* deletes temp file

* fix generated folder code

* make gofmt

* make gen-go

* make update-workspace

* add COPY apps/quotas to Dockerfile

* fix test mock

* fixes undefined NewFolderStatus()

* make gen-apps, and add func for NewFolderStatus

* make gen-apps again

* make update-workspace

* regen folder_object_gen.go

* gofmt

* fix linting

* apps/folder make update-workspace

* make gen-apps

* make gen-apps

* fixes enterprise_imports.go

* go get testcontainers

* adds feature toggle

* make update-workspace

* fix go mod

* fix another client mock

---------

Co-authored-by: Steve Simpson <steve@grafana.com>
2025-12-09 09:40:34 -06:00
Renato Costa 83311049ad fix: create dashboard in legacy storage within transaction (#114808)
fix: create dashboard within transaction
2025-12-09 10:16:33 -05:00
Mustafa Sencer Özcan 4c5d9cb95f feat: add unified storage data migration step for playlists (#114582)
* fix: add type

* feat: register step

* feat: add playlist support

* test: add test case

* fix: gen mock

* fix: go gen

* fix: lint

* fix: lint

* fix: tests

* fix: add resource

* fix: readd

* fix: address comments

* fix: independent playlist query for migrations

* fix: remove lock logic for sqlite

* fix: handle creation and update datetimes

* fix: query templating

* fix: simply resources and address comments
2025-12-04 15:15:00 +00:00
Mustafa Sencer Özcan 2f6836e78a fix: make graceful handle default for the malformed dashboard jsons during unified migration (#114295)
* fix: graceful handling by default

* fix: make fallback the default behavior
2025-11-24 14:12:52 +01:00
Denis Vodopianov 0e460a267e chore : Deprecating FeatureToggles.IsEnabled (#113062)
* Deprecating features.IsEnabled

* add one more nolint

* add one more nolint

* Give better hints to devs in the deprecation message of IsEnabledGlobally

* adding more doc strings

* fix linter after rebase

* Extend deprecation message
2025-11-21 18:43:42 +01:00
João Calisto c2c443757d Unified Storage: allow rebuilding indexes for resource with a new grpc endpoint (#113748)
* Unified Storage: allow rebuilding indexes for resource from a new grpc endpoint

* remove log line

* fix trace def

* lint

* fix after rebase

* addressing code review changes

* update with one channel per rebuild request

* other review suggestions

* update with review suggestions

* run mockery generate for MockResourceClient

* update tests

* update tests and lint

* fix test
2025-11-21 16:42:15 +00:00
Mustafa Sencer ÖzcanandRafael Paulovic 30c04ab3fc feat: inject unified data migrations in dual writer (#114138)
* feat: draft changes for on-prem unified migration

* feat: further draft changes for on-prem unified migration

* fix: remove some tbis

* refactor: rename

* fix: another approach

* fix: background service related issues

* fix: address comments

* fix: make gen-go

* fix: background service related issues

* feat: refactor dual writer and legacy migrator

* fix: minor issues

* feat: working version in oss

* fix: wire

* fix: revert test data override

* fix: enterprise related issues

* chore: add todo

* fix: revert dual writer method

* fix: lint

* chore: logger format

* fix: reduce log level

* fix: log change

* fix: disable

* fix: address comments

* fix: return error on dual writer service

* fix: merge conflict

---------

Co-authored-by: Rafael Paulovic <rafael.paulovic@grafana.com>
2025-11-20 16:40:20 +01:00
Mustafa Sencer Özcan 0d4ad01b65 feat: add unified data migrations for dashboard and folders (#113853) 2025-11-19 09:09:08 +01:00
owensmallwood 8dddff3ce4 Unified Storage: Pass ns, group, resource to GetResourceStats instead of just namespace (#114050)
* passes nsr to GetResourceStats instead of just namespace

* removes ns check

* fixes failing tests

* make update-workspace

* pass group and resource from rebuild request when getting resource stats
2025-11-18 13:05:21 -06:00
Jean-Philippe QuéménerandMustafa Sencer Özcan fd14d4a5ed feat(unified-storage): add tracing to dual writer and legacy storage (#113504)
Co-authored-by: Mustafa Sencer Özcan <32759850+mustafasencer@users.noreply.github.com>
2025-11-06 11:42:46 +00:00
Stephanie Hingtgen 891d5f0625 Unistore: Propagate DeprecatedLegacyID on upsert (#112619)
Unistore: Fix label propogation on upsert
2025-10-20 09:29:03 +02:00
Ryan McKinley 7c6afeabec Chore: Use IAM DBTime utility in dashboard migrator (#112589) 2025-10-17 17:48:30 +03:00
Stephanie Hingtgen bd7872c4d9 Dashboard History: Fix service account parsing (#112300) 2025-10-10 15:25:00 -05:00
Peter Štibraný d801b87db9 LastImportTime for resource. (#112153)
* LastImportTime for resource.

* Make StorageBackendImpl implement GetResourceLastImportTimes

* More missing implementations of GetResourceLastImportTimes

* Fix import.

* Skip TestGetResourceLastImportTime in TestBadgerKVStorageBackend.

* Implement GetResourceLastImportTimes by mockStorageBackend

* Bump test tolerance.

* Fix postgres query and timezone.

* Fix postgres query and timezone.

* Make linter happy.
2025-10-09 11:27:11 +02:00
Ryan McKinley 8f8e4a881a Dashboards: Avoid panic with invalid continue token (#111870) 2025-10-01 13:11:00 +02:00
Ivan Ortega AlbaIvan Ortega [ivanortegaalba@gmail.com](mailto:ivanortegaalba@gmail.com)Dominik Prokop [dominik.prokop@grafana.com](mailto:dominik.prokop@grafana.com)
a72e02f88a Fix dashboard migration discrepancies between backend and frontend implementations (use toEqual) (#110268)
**Highlights**

* **Single-version migrations**: add `targetVersion` to migrator & model, separate outputs, enforce exact version.
* **Datasource fixes**: include `apiVersion` in tests, empty-string → `{}`, preserve `{}` refs, drop unwanted defaults.
* **Panel defaults & nesting**: only top-level panels get defaults; preserve empty `transformations` context-aware; filter repeated panels.

* **Migration parity**

  * V16: collapsed rows, grid height parsing (`px`).
  * V17: omit `maxPerRow` when `minSpan=1`.
  * V19–V20: cleanup defaults (`targetBlank`, style).
  * V23–V24: template vars + table panel consistency.
  * V28: full singlestat/stat parity, mappings & color.
  * V30–V36: threshold logic, empty refs, nested targets.
* **Save-model cleanup**: replicate frontend defaults/filtering, drop null IDs, metadata, unused props.
* **Testing**: unified suites, dev dashboards (v42), full unit coverage for major migrations.

Co-authored-by: Ivan Ortega [ivanortegaalba@gmail.com](mailto:ivanortegaalba@gmail.com)
Co-authored-by: Dominik Prokop [dominik.prokop@grafana.com](mailto:dominik.prokop@grafana.com)
2025-09-24 12:20:25 +02:00
owensmallwood cace999671 Unified Storage: Add ListSinceModified to StorageBackend (#109697)
* WIP added ListSinceModified to StorageBackend interface

* fix compile time check

* Fix method name

* Fix naming

* fix the rest of the ListSinceModified names

* Uses resource key without name field

* get latest rv from resource_version. Update test.

* adds moar tests

* adds method stub for ListModifiedSince to other StorageBackend implementations

* adds dummy impl to noop storage backend for ListModifiedSince

* skip tests for badger kv backend for now

* fixes tests and adds badgerkv impl for ListModifiedSince

* add badger kv impl

* adds test for new query

* adds test data for new query

* adds ListModifiedSince stub to mockStorageBackend

* uncomment tests

* refactors ListModifiedSince to return an iter.seq2 and handles deduplication. Updates tests. Updates query result sorting.

* remove comments

* remove folder from query (dont need it, yet?)

* regen test queries

* updates test

* updates function comment

* use resourcepb.ResourceKey instead of ModifiedResourceKey

* wrap seq in single transaction. Rollback transaction after 30s if iterator never used. Only track last seen event. Formatting.

* skip TestListModifiedSince for kv backend

* use WatchEvent_Type for action type

* remove redundant fields from order by clause and regen test data for query

* remove redundant fields from order by clause and regen test data for query
2025-08-20 11:54:31 -06:00
maiconandJean-Philippe Quéméner 5270d8e53f fix(unified-storage): remove permissions after executing on both stores (#109722)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
Co-authored-by: Jean-Philippe Quéméner <jeanphilippe.quemener@grafana.com>
2025-08-15 15:57:42 +00:00
Mustafa Sencer ÖzcanandWill Assis 98e37f2ca9 fix: special handling of unmarshalling for invalid json dashboards (#108519)
* implement special dashboard fallback logic when dashboard data has invalid json behind feature flag

---------

Co-authored-by: Will Assis <william@williamassis.com>
2025-07-30 10:18:38 -04:00
Bruno Abrantes a81c28b153 chore: include stackId when marking a dashboard as rejected during migration (#108887)
Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-07-30 08:37:10 +00:00
Will Assis 71cb623cb7 improve logging and fail unified-storage migration with more than 0 errors (#108471)
improve logging and fail unified-storage migration with more than 0 errors
2025-07-22 13:44:04 +00:00
Bruno Abrantes be79eb0ace fix: handle dashboards without version entries in unified storage migration (#108139)
* fix: handle dashboards without version entries in migration

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

* fix: adds AllowFallback flag to enable COALESCE logic explicitly during migrations

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

* fix: update dependencies

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

* fix: maintain original behavior when not using AllowFallback option

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

---------

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-07-21 15:32:24 +02:00
Bruno Abrantes c06c1b1e8a fix: Makes created and updated by nullable strings for library panel migration (#108122)
Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-07-15 16:30:31 +02:00
Ryan McKinley 3f502f305d Chore: Update mocks with recent mockery (#107816) 2025-07-09 09:15:34 +02:00
Stephanie Hingtgen b1d3155b60 K8s: Library Panels: Add rerouting for get (#107362) 2025-06-30 13:26:24 -05:00
Mustafa Sencer Özcan c47fc53eee fix: set withHistory flag to true for non-interactive mode (#107263) 2025-06-26 16:08:30 +00:00
Mustafa Sencer Özcan 3a13c2db64 chore: add logging around migration cli (#107046) 2025-06-20 15:20:06 +00:00
Stephanie Hingtgen feeced9618 K8s: Dashboards /apis: Fix library element connections (#106734) 2025-06-13 21:40:39 +02:00
Georges Chaudy 04d39cbbc6 unistore: split ListIterator and ListHistory in StorageBackend (#105654)
split listIterator from ListHistory
2025-05-23 15:00:18 +02:00
Ryan McKinley d896f31c43 Dashboards: report valid folder count (#105484) 2025-05-16 14:51:04 +03:00
Jean-Philippe Quéméner 002f46736a refactor(unified-storage): move generated protos to own pkg (#105356) 2025-05-15 21:36:52 +02:00
Ryan McKinley da24ad06bd Provisioning: Hide full path for classic provisioing resources (#105440) 2025-05-15 16:01:12 +00:00
Stephanie Hingtgen 9968576acf Folders & Dashboards: Cleanup timestamps and error codes (#104665)
K8s: Fix timestamps and error codes
2025-04-30 03:03:21 +03:00
Stephanie Hingtgen b5281c923f K8s: Dashboards: Fix provisioned dashboard cleanup (#104504) 2025-04-25 05:55:50 -06:00
Ryan McKinley b09d79b21c K8s/Dashboard: Promote from alpha1 to beta1 (#104009) 2025-04-23 20:54:35 +03:00
b887e8aa05 K8s: Dashboards: Add fine grained access control checks to /apis (#104347)
---------

Co-authored-by: Ieva <ieva.vasiljeva@grafana.com>
Co-authored-by: Gabriel MABILLE <gabriel.mabille@grafana.com>
Co-authored-by: Marco de Abreu <marco.deabreu@grafana.com>
Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-04-23 03:29:05 +01:00
Matheus Macabu fc9f32a9f6 SQLTemplates: Add helper to ensure all templates have a test-case (#103964)
* SQLTemplates: Add helper to ensure all templates have a test-case associated

* UnifiedStorage: Add missing sql template test case

* LegacyDashboards: Add sql templates fs to test cases for exhaustiveness check

* RBACStore: Add sql templates fs to test cases for exhaustiveness check

* LegacyIAM: Add missing sql template test cases
2025-04-22 11:21:51 +02:00
Marco de AbreuandMarco de Abreu d57fa1291f Dashboards: Disallow List with RV filter on legacy & align RV format (#104003)
Throw errors when trying to call List in legacy with an RV specified
Change ResourceVersion to microseconds

Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
2025-04-15 22:08:54 +01:00
Ryan McKinley 0283c98e30 K8s/Folders: Use v1beta1 and app-sdk based spec (#103975) 2025-04-14 23:20:10 +03:00
c47ab101d1 Dashboards: Add Dashboard Schema validation (2) (#103844)
* Activate schema validation and align underlying systems

* update to save as v0 if not the right schema version

* Resolve merge conflicts

* Move RequireApiErrorStatus to tests package

* Add mutation tests

* Fix lint

* Only do min version check if dashboard is v1

* Fix lint and disable provisioning test

* Revert provisioning changes

* Revert more tests and add schema test

* Reran gen

* SQL Dashboard save

* Adjust APIVERSION

* Fixed mutation test

* Add logging on downgrade

---------

Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-04-11 23:05:41 +02:00
Ryan McKinley b39eaac69e Unistore: Keep apiVersion from the legacy SQL table (#103939)
keep apiversion
2025-04-11 22:49:45 +03:00
Stephanie Hingtgen f5ad1ef69b K8s: Folders: Add v1 api (#103842) 2025-04-11 13:09:52 +01:00
Marco de Abreu b2f80d4dca Dashboards: Add Dashboard API Validation tests and fix underlying issues (#103502) 2025-04-07 05:44:28 +03:00
Roberto Jiménez Sánchez 364247e26b Provisioning: add most mocks to unit test provisioning jobs (#103446)
* Use interface for factories

* Add most mocks to unit test jobs

* Generate more mocks for main jobs package

* Interface for parser

* Use clients directly without parser

* Regenerate mock

* Eliminate resource clients from Parser

* Interface for folder tree
2025-04-04 15:42:20 +03:00
Stephanie Hingtgen 4918d8720c Dashboard Restore: Remove experimental functionality under feature flag dashboardRestore for now - this will be reworked (#103204) 2025-04-03 02:52:54 -05:00