Mustafa Sencer Özcan
2d6c1c4e9e
* docs: add documentation for unified storage migrations * docs: move * docs: rename title * docs: add docs * fix: update table * fix: lint * docs: add migration table explanation
123 lines
4.0 KiB
Markdown
123 lines
4.0 KiB
Markdown
# Unified storage data migrations
|
|
|
|
Automated migration system for moving Grafana resources from legacy SQL storage to unified storage.
|
|
|
|
## Overview
|
|
|
|
The migration system transfers resources from legacy SQL tables to Grafana's unified storage backend. It runs automatically during Grafana startup and validates data integrity after each migration.
|
|
|
|
### Supported resources
|
|
|
|
| Resource | API Group | Legacy table |
|
|
|----------|-----------|--------------|
|
|
| Folders | `folder.grafana.app` | `dashboard` |
|
|
| Dashboards | `dashboard.grafana.app` | `dashboard` |
|
|
| Library panels | `dashboard.grafana.app` | `library_element` |
|
|
| Playlists | `playlist.grafana.app` | `playlist` |
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ ResourceMigration │
|
|
│ (Orchestrates per-organization migration) │
|
|
└──────────────────────────┬──────────────────────────────────┘
|
|
│
|
|
┌───────────────────┼───────────────────┐
|
|
▼ ▼ ▼
|
|
UnifiedMigrator Validators BulkProcess API
|
|
(Stream legacy (Validate after (Write to unified
|
|
resources) migration) storage)
|
|
```
|
|
|
|
### Components
|
|
|
|
- **`service.go`**: Migration service entry point and registration
|
|
- **`migrator.go`**: Core migration logic using streaming BulkProcess API
|
|
- **`resource_migration.go`**: Per-organization migration execution
|
|
- **`validator.go`**: Post-migration validation (CountValidator, FolderTreeValidator)
|
|
- **`resources.go`**: Registry of migratable resource types
|
|
|
|
## How migrations work
|
|
|
|
### Migration flow
|
|
|
|
1. Grafana starts and checks migration status in `unifiedstorage_migration_log` table
|
|
2. For each organization, the migrator:
|
|
- Reads resources from legacy SQL tables
|
|
- Streams resources to unified storage via BulkProcess API
|
|
- Runs validators to verify data integrity
|
|
3. Records migration result in `unifiedstorage_migration_log` table
|
|
|
|
### Per-organization execution
|
|
|
|
Migrations run independently for each organization using namespace format `org-{orgId}`.
|
|
|
|
## Validators
|
|
|
|
### CountValidator
|
|
|
|
Compares resource counts between legacy SQL and unified storage. Accounts for rejected items during validation.
|
|
|
|
### FolderTreeValidator
|
|
|
|
Verifies folder parent-child relationships are preserved after migration.
|
|
|
|
## Configuration
|
|
|
|
To enable migrations, set the following in your Grafana configuration:
|
|
|
|
```ini
|
|
[unified_storage]
|
|
disable_data_migrations = false
|
|
```
|
|
|
|
## Monitoring
|
|
|
|
### Log messages
|
|
|
|
Successful migration:
|
|
|
|
```
|
|
info: storage.unified.resource_migration Starting migration for all organizations
|
|
info: storage.unified.resource_migration Migration completed successfully for all organizations
|
|
```
|
|
|
|
Failed migration:
|
|
|
|
```
|
|
error: storage.unified.resource_migration Migration validation failed
|
|
```
|
|
|
|
### Migration status
|
|
|
|
Query the migration log table to check status:
|
|
|
|
```sql
|
|
SELECT * FROM unifiedstorage_migration_log WHERE migration_id LIKE '%folders-dashboards%';
|
|
```
|
|
|
|
The `migration_id` is defined in `service.go` during registration. Ideally, it should be the resource type(s) being migrated.
|
|
|
|
## Development
|
|
|
|
### Adding a new validator
|
|
|
|
Implement the `Validator` interface:
|
|
|
|
```go
|
|
type Validator interface {
|
|
Name() string
|
|
Validate(ctx context.Context, sess *xorm.Session, response *resourcepb.BulkResponse, log log.Logger) error
|
|
}
|
|
```
|
|
|
|
Register the validator in `service.go` when creating the `ResourceMigration`.
|
|
|
|
### Adding a new resource type
|
|
|
|
1. Add the resource definition to `registeredResources` in `resources.go`
|
|
2. Implement the migrator function in the `MigrationDashboardAccessor` interface
|
|
3. Register the migration in `service.go`
|
|
|