grafana/pkg/storage/unified/migrations

Unified storage data migrations

Automated migration system for moving Grafana resources from legacy SQL storage to unified storage.

Overview

The migration system transfers resources from legacy SQL tables to Grafana's unified storage backend. It runs automatically during Grafana startup and validates data integrity after each migration.

Supported resources

Resource	API Group	Legacy table
Folders	folder.grafana.app	dashboard
Dashboards	dashboard.grafana.app	dashboard
Library panels	dashboard.grafana.app	library_element
Playlists	playlist.grafana.app	playlist

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    ResourceMigration                        │
│        (Orchestrates per-organization migration)            │
└──────────────────────────┬──────────────────────────────────┘
                           │
       ┌───────────────────┼───────────────────┐
       ▼                   ▼                   ▼
  UnifiedMigrator      Validators         BulkProcess API
  (Stream legacy     (Validate after      (Write to unified
   resources)         migration)           storage)

Components

• service.go: Migration service entry point and registration
• migrator.go: Core migration logic using streaming BulkProcess API
• resource_migration.go: Per-organization migration execution
• validator.go: Post-migration validation (CountValidator, FolderTreeValidator)
• resources.go: Registry of migratable resource types

How migrations work

Migration flow

1. Grafana starts and checks migration status in unifiedstorage_migration_log table
2. For each organization, the migrator:
   • Reads resources from legacy SQL tables
   • Streams resources to unified storage via BulkProcess API
   • Runs validators to verify data integrity
3. Records migration result in unifiedstorage_migration_log table

Per-organization execution

Migrations run independently for each organization using namespace format org-{orgId}.

Validators

CountValidator

Compares resource counts between legacy SQL and unified storage. Accounts for rejected items during validation.

FolderTreeValidator

Verifies folder parent-child relationships are preserved after migration.

Configuration

To enable migrations, set the following in your Grafana configuration:

[unified_storage]
disable_data_migrations = false

Monitoring

Log messages

Successful migration:

info: storage.unified.resource_migration Starting migration for all organizations
info: storage.unified.resource_migration Migration completed successfully for all organizations

Failed migration:

error: storage.unified.resource_migration Migration validation failed

Migration status

Query the migration log table to check status:

SELECT * FROM unifiedstorage_migration_log WHERE migration_id LIKE '%folders-dashboards%';

The migration_id is defined in service.go during registration. Ideally, it should be the resource type(s) being migrated.

Development

Adding a new validator

Implement the Validator interface:

type Validator interface {
    Name() string
    Validate(ctx context.Context, sess *xorm.Session, response *resourcepb.BulkResponse, log log.Logger) error
}

Register the validator in service.go when creating the ResourceMigration.

Adding a new resource type

1. Add the resource definition to registeredResources in resources.go
2. Implement the migrator function in the MigrationDashboardAccessor interface
3. Register the migration in service.go